[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:50:14 dist-test-slave-pww0 krb5kdc[28271](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:50:14 dist-test-slave-pww0 krb5kdc[28271](info): set up 2 sockets
Aug 10 03:50:14 dist-test-slave-pww0 krb5kdc[28271](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 03:50:16.450146 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.543s user 0.009s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:50:16 dist-test-slave-pww0 krb5kdc[28271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:50:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:50:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:50:16.591445 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:38183
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:45821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:38183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:16.728672 28288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:16.729038 28288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:16.729105 28288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:16.734293 28288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:50:16.734392 28288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:16.734412 28288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:16.734429 28288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:50:16.734443 28288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:50:16.741807 28288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:45821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:38183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:38183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:16.743479 28288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:16.744966 28288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:16.760849 28307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:16.760923 28288 server_base.cc:1034] running on GCE node
W20260810 03:50:16.760979 28308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:16.760852 28311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:16.762089 28288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:16.764148 28288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:16.765398 28288 hybrid_clock.cc:648] HybridClock initialized: now 1786333816765339 us; error 81 us; skew 500 ppm
Aug 10 03:50:16 dist-test-slave-pww0 krb5kdc[28271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333816, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:16.769559 28288 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:50:16.770979 28288 webserver.cc:460] Webserver started at http://127.27.153.126:37503/ using document root <none> and password file <none>
I20260810 03:50:16.772137 28288 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:16.772220 28288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:16.772713 28288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:16.776118 28288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "c622f16430df4f4a9a58fa3bd11a013f"
format_stamp: "Formatted at 2026-08-10 03:50:16 on dist-test-slave-pww0"
server_key: "7ccf1b684a1d381044255fb15b95b81f"
server_key_iv: "bc0988d5b73653c78a4d59b62255d6dd"
server_key_version: "encryptionkey@0"
I20260810 03:50:16.776868 28288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "c622f16430df4f4a9a58fa3bd11a013f"
format_stamp: "Formatted at 2026-08-10 03:50:16 on dist-test-slave-pww0"
server_key: "7ccf1b684a1d381044255fb15b95b81f"
server_key_iv: "bc0988d5b73653c78a4d59b62255d6dd"
server_key_version: "encryptionkey@0"
I20260810 03:50:16.783010 28288 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 03:50:16.787413 28323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:16.789325 28288 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:50:16.789564 28288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "c622f16430df4f4a9a58fa3bd11a013f"
format_stamp: "Formatted at 2026-08-10 03:50:16 on dist-test-slave-pww0"
server_key: "7ccf1b684a1d381044255fb15b95b81f"
server_key_iv: "bc0988d5b73653c78a4d59b62255d6dd"
server_key_version: "encryptionkey@0"
I20260810 03:50:16.789701 28288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:16.823823 28288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:16.828840 28288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:16.829212 28288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:16.840955 28288 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:38183
I20260810 03:50:16.841022 28418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:38183 every 8 connection(s)
I20260810 03:50:16.842568 28288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:50:16.848582 28419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:16.851554 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 28288
I20260810 03:50:16.851766 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:50:16.853963 28261 external_mini_cluster.cc:1428] Setting key 56e531426037123a6e0f759b71bf9235
I20260810 03:50:16.861123 28419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f: Bootstrap starting.
I20260810 03:50:16.865835 28419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:16.867498 28419 log.cc:824] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:16.870826 28419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f: No bootstrap required, opened a new log
I20260810 03:50:16.875864 28419 raft_consensus.cc:348] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c622f16430df4f4a9a58fa3bd11a013f" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 38183 } }
I20260810 03:50:16.876214 28419 raft_consensus.cc:374] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:16.876322 28419 raft_consensus.cc:729] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c622f16430df4f4a9a58fa3bd11a013f, State: Initialized, Role: FOLLOWER
I20260810 03:50:16.877143 28419 consensus_queue.cc:260] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c622f16430df4f4a9a58fa3bd11a013f" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 38183 } }
I20260810 03:50:16.877336 28419 raft_consensus.cc:388] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:50:16.877471 28419 raft_consensus.cc:482] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:50:16.877621 28419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f [term 0 FOLLOWER]: Advancing to term 1
Aug 10 03:50:16 dist-test-slave-pww0 krb5kdc[28271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:16.878937 28419 raft_consensus.cc:504] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c622f16430df4f4a9a58fa3bd11a013f" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 38183 } }
I20260810 03:50:16.879698 28419 leader_election.cc:302] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c622f16430df4f4a9a58fa3bd11a013f; no voters:
I20260810 03:50:16.880129 28419 leader_election.cc:288] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:50:16.880285 28425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:16.880471 28425 raft_consensus.cc:686] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f [term 1 LEADER]: Becoming Leader. State: Replica: c622f16430df4f4a9a58fa3bd11a013f, State: Running, Role: LEADER
I20260810 03:50:16.880904 28425 consensus_queue.cc:237] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c622f16430df4f4a9a58fa3bd11a013f" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 38183 } }
I20260810 03:50:16.882056 28419 sys_catalog.cc:564] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:50:16.884055 28430 sys_catalog.cc:455] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c622f16430df4f4a9a58fa3bd11a013f. Latest consensus state: current_term: 1 leader_uuid: "c622f16430df4f4a9a58fa3bd11a013f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c622f16430df4f4a9a58fa3bd11a013f" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 38183 } } }
I20260810 03:50:16.884308 28430 sys_catalog.cc:458] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:16.884725 28429 sys_catalog.cc:455] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c622f16430df4f4a9a58fa3bd11a013f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c622f16430df4f4a9a58fa3bd11a013f" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 38183 } } }
I20260810 03:50:16.884845 28429 sys_catalog.cc:458] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:16.888298 28422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:16.860261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56728 (local address 127.27.153.126:38183)
0810 03:50:16.861007 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:16.861024 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:16.863722 (+ 2698us) server_negotiation.cc:407] Connection header received
0810 03:50:16.867580 (+ 3858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:16.867610 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:16.868221 (+ 611us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:16.868615 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:16.871369 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:16.872473 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:16.873884 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:16.874468 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:16.874641 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:16.879260 (+ 4619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:16.879316 (+ 56us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:16.879333 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:16.879376 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:16.881543 (+ 2167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:16.882195 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:16.882203 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:16.882209 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:16.882362 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:16.882769 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:16.882772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:16.882774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:16.883631 (+ 857us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:16.883869 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:16.886681 (+ 2812us) server_negotiation.cc:299] Negotiation successful
0810 03:50:16.887059 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":281,"threads_started":1}
W20260810 03:50:16.890676 28454 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:50:16.890821 28454 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:50:16.890905 28452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:50:16.892079 28452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:50:16.901008 28452 catalog_manager.cc:1347] Generated new cluster ID: f0e78b89346147278a172191fe480c2b
I20260810 03:50:16.901127 28452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:50:16.911502 28452 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:50:16.912639 28452 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:50:16.924144 28452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c622f16430df4f4a9a58fa3bd11a013f: Generated new TSK 0
I20260810 03:50:16.925262 28452 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.153.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
I20260810 03:50:16.970939 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:38183
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:45821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:17.117435 28461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:17.117769 28461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:17.117817 28461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:17.121959 28461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:17.122043 28461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:17.122115 28461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:50:17.129729 28461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:45821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:38183
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:17.131248 28461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:17.132540 28461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:17.143638 28466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:17.143639 28471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:17.143638 28469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:17.143918 28461 server_base.cc:1034] running on GCE node
I20260810 03:50:17.144450 28461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:17.145165 28461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:17.146344 28461 hybrid_clock.cc:648] HybridClock initialized: now 1786333817146324 us; error 33 us; skew 500 ppm
Aug 10 03:50:17 dist-test-slave-pww0 krb5kdc[28271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333817, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:17.149421 28461 init.cc:377] Logged in from keytab as kudu/127.27.153.65@KRBTEST.COM (short username kudu)
I20260810 03:50:17.150421 28461 webserver.cc:460] Webserver started at http://127.27.153.65:33767/ using document root <none> and password file <none>
I20260810 03:50:17.150982 28461 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:17.151053 28461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:17.151278 28461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:17.153446 28461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "525ea6022300467fbec284c3eca8e90d"
format_stamp: "Formatted at 2026-08-10 03:50:17 on dist-test-slave-pww0"
server_key: "6debb15fdc5cc3875d4597a5fa7bfb19"
server_key_iv: "87a646da2d67497e8ab80650b9aa5885"
server_key_version: "encryptionkey@0"
I20260810 03:50:17.154039 28461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "525ea6022300467fbec284c3eca8e90d"
format_stamp: "Formatted at 2026-08-10 03:50:17 on dist-test-slave-pww0"
server_key: "6debb15fdc5cc3875d4597a5fa7bfb19"
server_key_iv: "87a646da2d67497e8ab80650b9aa5885"
server_key_version: "encryptionkey@0"
I20260810 03:50:17.158344 28461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:50:17.161083 28480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:17.162413 28461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:50:17.162547 28461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "525ea6022300467fbec284c3eca8e90d"
format_stamp: "Formatted at 2026-08-10 03:50:17 on dist-test-slave-pww0"
server_key: "6debb15fdc5cc3875d4597a5fa7bfb19"
server_key_iv: "87a646da2d67497e8ab80650b9aa5885"
server_key_version: "encryptionkey@0"
I20260810 03:50:17.162659 28461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:17.175426 28461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:17.178664 28461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:17.178869 28461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:17.179648 28461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:17.181007 28461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:17.181066 28461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:17.181128 28461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:17.181152 28461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:17.191092 28461 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:46275
I20260810 03:50:17.191118 28688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:46275 every 8 connection(s)
I20260810 03:50:17.192123 28461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:50:17.201128 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 28461
I20260810 03:50:17.201256 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:50:17.201515 28261 external_mini_cluster.cc:1428] Setting key 47c19b75f676e9ad776fbd8fd051d133
Aug 10 03:50:17 dist-test-slave-pww0 krb5kdc[28271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333817, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:17.208039 28422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:17.194507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:57007 (local address 127.27.153.126:38183)
0810 03:50:17.194686 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:17.194690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:17.195758 (+ 1068us) server_negotiation.cc:407] Connection header received
0810 03:50:17.197520 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:17.197523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:17.197588 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:17.197663 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:17.198971 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:17.199561 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:17.200760 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:17.201076 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:17.201193 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:17.204353 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:17.204356 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:17.204358 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:17.204385 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:17.205775 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:17.206327 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:17.206331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:17.206332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:17.206421 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:17.206779 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:17.206782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:17.206783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:17.206989 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:17.207081 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:17.207687 (+ 606us) server_negotiation.cc:299] Negotiation successful
0810 03:50:17.207863 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:50:17.208945 28694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:17.194983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:38183 (local address 127.27.153.65:57007)
0810 03:50:17.195595 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:17.195646 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:17.197272 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:17.197864 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:17.197871 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:17.198270 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:17.198806 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:17.198822 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:17.199695 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:17.199698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:17.200632 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:17.200642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:17.201286 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:17.201292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:17.201494 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:17.202256 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:17.202277 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:17.204170 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:17.205905 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:17.205912 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:17.205932 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:17.206221 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:17.206510 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:17.206513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:17.206516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:17.206676 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:17.207098 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:17.207106 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:17.207403 (+ 297us) client_negotiation.cc:769] Sending connection context
0810 03:50:17.207625 (+ 222us) client_negotiation.cc:240] Negotiation successful
0810 03:50:17.208006 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":133,"threads_started":1}
I20260810 03:50:17.210669 28692 heartbeater.cc:347] Connected to a master server at 127.27.153.126:38183
I20260810 03:50:17.210956 28692 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:17.211608 28692 heartbeater.cc:511] Master 127.27.153.126:38183 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.153.66@KRBTEST.COM; defaulting to no policy
I20260810 03:50:17.214314 28358 ts_manager.cc:194] Registered new tserver with Master: 525ea6022300467fbec284c3eca8e90d (127.27.153.65:46275)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.66@KRBTEST.COM" created.
I20260810 03:50:17.216218 28358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:57007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
I20260810 03:50:17.232729 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:38183
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:45821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:17.350932 28698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:17.351229 28698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:17.351280 28698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:17.355507 28698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:17.355595 28698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:17.355683 28698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:50:17.361315 28698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:45821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:38183
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:17.362442 28698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:17.363531 28698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:17.373834 28707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:17.373984 28698 server_base.cc:1034] running on GCE node
W20260810 03:50:17.373849 28705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:17.373821 28710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:17.374470 28698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:17.375187 28698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:17.376390 28698 hybrid_clock.cc:648] HybridClock initialized: now 1786333817376352 us; error 48 us; skew 500 ppm
Aug 10 03:50:17 dist-test-slave-pww0 krb5kdc[28271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333817, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:17.379316 28698 init.cc:377] Logged in from keytab as kudu/127.27.153.66@KRBTEST.COM (short username kudu)
I20260810 03:50:17.380523 28698 webserver.cc:460] Webserver started at http://127.27.153.66:41975/ using document root <none> and password file <none>
I20260810 03:50:17.381104 28698 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:17.381176 28698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:17.381394 28698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:17.383695 28698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "61adcd569b0f4be89baa9f1b2c8e276d"
format_stamp: "Formatted at 2026-08-10 03:50:17 on dist-test-slave-pww0"
server_key: "68c7753a5f82f5b1ec9678e3f55d2133"
server_key_iv: "2e5fac4c47d3ff0ac47bda93ba8dc66d"
server_key_version: "encryptionkey@0"
I20260810 03:50:17.384384 28698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "61adcd569b0f4be89baa9f1b2c8e276d"
format_stamp: "Formatted at 2026-08-10 03:50:17 on dist-test-slave-pww0"
server_key: "68c7753a5f82f5b1ec9678e3f55d2133"
server_key_iv: "2e5fac4c47d3ff0ac47bda93ba8dc66d"
server_key_version: "encryptionkey@0"
I20260810 03:50:17.388649 28698 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:50:17.391306 28720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:17.392740 28698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:17.392876 28698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "61adcd569b0f4be89baa9f1b2c8e276d"
format_stamp: "Formatted at 2026-08-10 03:50:17 on dist-test-slave-pww0"
server_key: "68c7753a5f82f5b1ec9678e3f55d2133"
server_key_iv: "2e5fac4c47d3ff0ac47bda93ba8dc66d"
server_key_version: "encryptionkey@0"
I20260810 03:50:17.392997 28698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:17.409945 28698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:17.413516 28698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:17.413727 28698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:17.414382 28698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:17.415288 28698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:17.415342 28698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:17.415400 28698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:17.415427 28698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:17.425283 28698 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:34061
I20260810 03:50:17.425308 28929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:34061 every 8 connection(s)
I20260810 03:50:17.426242 28698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:50:17.430574 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 28698
I20260810 03:50:17.430667 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:50:17.430902 28261 external_mini_cluster.cc:1428] Setting key 42ed5f1075a8df9bc6bc52c9df770b19
Aug 10 03:50:17 dist-test-slave-pww0 krb5kdc[28271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333817, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.153.67@KRBTEST.COM; defaulting to no policy
I20260810 03:50:17.440629 28422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:17.428281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:33465 (local address 127.27.153.126:38183)
0810 03:50:17.428424 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:17.428428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:17.429360 (+ 932us) server_negotiation.cc:407] Connection header received
0810 03:50:17.430819 (+ 1459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:17.430822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:17.430881 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:17.430960 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:17.432406 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:17.432934 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:17.434090 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:17.434393 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:17.434494 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:17.437158 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:17.437162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:17.437164 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:17.437190 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:17.438553 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:17.439076 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:17.439080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:17.439081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:17.439159 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:17.439505 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:17.439508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:17.439510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:17.439709 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:17.439800 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:17.440322 (+ 522us) server_negotiation.cc:299] Negotiation successful
0810 03:50:17.440457 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:50:17.441531 28934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:17.428673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:38183 (local address 127.27.153.66:33465)
0810 03:50:17.429185 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:17.429234 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:17.430584 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:17.431150 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:17.431157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:17.431694 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:17.432259 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:17.432281 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:17.433059 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:17.433062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:17.433961 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:17.433971 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:17.434545 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:17.434549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:17.434730 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:17.435302 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:17.435319 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:17.436966 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:17.438674 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:17.438681 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:17.438699 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:17.438971 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:17.439244 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:17.439247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:17.439250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:17.439392 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:17.439814 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:17.439821 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:17.440083 (+ 262us) client_negotiation.cc:769] Sending connection context
0810 03:50:17.440263 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 03:50:17.440624 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":121,"threads_started":1}
I20260810 03:50:17.442876 28931 heartbeater.cc:347] Connected to a master server at 127.27.153.126:38183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.67@KRBTEST.COM" created.
I20260810 03:50:17.443130 28931 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:17.443806 28931 heartbeater.cc:511] Master 127.27.153.126:38183 requested a full tablet report, sending...
I20260810 03:50:17.445037 28358 ts_manager.cc:194] Registered new tserver with Master: 61adcd569b0f4be89baa9f1b2c8e276d (127.27.153.66:34061)
I20260810 03:50:17.445667 28358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:33465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
I20260810 03:50:17.461213 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:38183
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:45821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:17.582455 28938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:17.582762 28938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:17.582810 28938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:17.587193 28938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:17.587278 28938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:17.587343 28938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:50:17.593117 28938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:45821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:38183
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:17.594336 28938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:17.595588 28938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:17.605923 28943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:17.605952 28945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:17.606069 28938 server_base.cc:1034] running on GCE node
W20260810 03:50:17.605957 28954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:17.606765 28938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:17.607492 28938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:17.608703 28938 hybrid_clock.cc:648] HybridClock initialized: now 1786333817608679 us; error 37 us; skew 500 ppm
Aug 10 03:50:17 dist-test-slave-pww0 krb5kdc[28271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333817, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:17.611821 28938 init.cc:377] Logged in from keytab as kudu/127.27.153.67@KRBTEST.COM (short username kudu)
I20260810 03:50:17.612833 28938 webserver.cc:460] Webserver started at http://127.27.153.67:35489/ using document root <none> and password file <none>
I20260810 03:50:17.613427 28938 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:17.613498 28938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:17.613722 28938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:17.615984 28938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "da62ce2beeea49a5a9d9ac095ad23f5d"
format_stamp: "Formatted at 2026-08-10 03:50:17 on dist-test-slave-pww0"
server_key: "6f9985657bdbad9b4ebff762671d12ec"
server_key_iv: "a2ccce037239a4f1eb217a456c277e40"
server_key_version: "encryptionkey@0"
I20260810 03:50:17.616587 28938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "da62ce2beeea49a5a9d9ac095ad23f5d"
format_stamp: "Formatted at 2026-08-10 03:50:17 on dist-test-slave-pww0"
server_key: "6f9985657bdbad9b4ebff762671d12ec"
server_key_iv: "a2ccce037239a4f1eb217a456c277e40"
server_key_version: "encryptionkey@0"
I20260810 03:50:17.620651 28938 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:50:17.623195 28964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:17.624563 28938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:50:17.624704 28938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "da62ce2beeea49a5a9d9ac095ad23f5d"
format_stamp: "Formatted at 2026-08-10 03:50:17 on dist-test-slave-pww0"
server_key: "6f9985657bdbad9b4ebff762671d12ec"
server_key_iv: "a2ccce037239a4f1eb217a456c277e40"
server_key_version: "encryptionkey@0"
I20260810 03:50:17.624825 28938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:17.637246 28938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:17.640810 28938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:17.641013 28938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:17.641680 28938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:17.642603 28938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:17.642657 28938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:17.642714 28938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:17.642742 28938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:17.652109 28938 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:39247
I20260810 03:50:17.652145 29174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:39247 every 8 connection(s)
I20260810 03:50:17.653087 28938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:50:17.659741 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 28938
I20260810 03:50:17.659886 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:50:17.660175 28261 external_mini_cluster.cc:1428] Setting key 45b3af4f51f187b16495dd484d3738c6
Aug 10 03:50:17 dist-test-slave-pww0 krb5kdc[28271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333817, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:17.668972 28422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:17.655047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:55781 (local address 127.27.153.126:38183)
0810 03:50:17.655206 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:17.655211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:17.656242 (+ 1031us) server_negotiation.cc:407] Connection header received
0810 03:50:17.657923 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:17.657926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:17.657985 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:17.658070 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:17.659491 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:17.660293 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:17.661761 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:17.662022 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:17.662106 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:17.664971 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:17.664974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:17.664976 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:17.665015 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:17.666533 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:17.667234 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:17.667237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:17.667239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:17.667345 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:17.667790 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:17.667793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:17.667794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:17.668014 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:17.668115 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:17.668724 (+ 609us) server_negotiation.cc:299] Negotiation successful
0810 03:50:17.668843 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:50:17.669907 29178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:17.655520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:38183 (local address 127.27.153.67:55781)
0810 03:50:17.656061 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:17.656113 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:17.657648 (+ 1535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:17.658276 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:17.658284 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:17.658689 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:17.659243 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:17.659261 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:17.660442 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:17.660446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:17.661643 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:17.661656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:17.662185 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:17.662191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:17.662413 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:17.663185 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:17.663203 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:17.664780 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:17.666675 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:17.666687 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:17.666709 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:17.667096 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:17.667448 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:17.667451 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:17.667452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:17.667686 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:17.668161 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:17.668170 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:17.668456 (+ 286us) client_negotiation.cc:769] Sending connection context
0810 03:50:17.668658 (+ 202us) client_negotiation.cc:240] Negotiation successful
0810 03:50:17.668966 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":137,"threads_started":1}
I20260810 03:50:17.671175 29175 heartbeater.cc:347] Connected to a master server at 127.27.153.126:38183
I20260810 03:50:17.671424 29175 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:17.671957 29175 heartbeater.cc:511] Master 127.27.153.126:38183 requested a full tablet report, sending...
I20260810 03:50:17.673050 28358 ts_manager.cc:194] Registered new tserver with Master: da62ce2beeea49a5a9d9ac095ad23f5d (127.27.153.67:39247)
I20260810 03:50:17.673666 28358 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:55781
I20260810 03:50:17.675531 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 03:50:17 dist-test-slave-pww0 krb5kdc[28271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333817, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 03:50:17 dist-test-slave-pww0 krb5kdc[28271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333817, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.153.65@KRBTEST.COM
I20260810 03:50:17.704704 29184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:17.695023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41940 (local address 127.27.153.65:46275)
0810 03:50:17.695378 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:17.695393 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:17.695421 (+ 28us) server_negotiation.cc:407] Connection header received
0810 03:50:17.695533 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:17.695541 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:17.695751 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:17.695877 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:17.696575 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:17.697476 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:17.698385 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:17.698781 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:17.698865 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:17.701251 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:17.701262 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:17.701277 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:17.701304 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:17.702860 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:17.703307 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:17.703312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:17.703314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:17.703391 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:17.703697 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:17.703699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:17.703701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:17.703962 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:17.704130 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:17.704350 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 03:50:17.704500 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":81,"threads_started":1}
W20260810 03:50:17.705775 28532 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41940
Aug 10 03:50:17 dist-test-slave-pww0 krb5kdc[28271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 03:50:17 dist-test-slave-pww0 krb5kdc[28271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.65@KRBTEST.COM
I20260810 03:50:17.733776 29184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:17.724699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41942 (local address 127.27.153.65:46275)
0810 03:50:17.724845 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:17.724849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:17.725014 (+ 165us) server_negotiation.cc:407] Connection header received
0810 03:50:17.725113 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:17.725116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:17.725166 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:17.725250 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:17.726013 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:17.726576 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:17.727445 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:17.727780 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:17.727874 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:17.730536 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:17.730539 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:17.730541 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:17.730575 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:17.732060 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:17.732533 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:17.732536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:17.732538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:17.732630 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:17.732896 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:17.732900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:17.732901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:17.733087 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:17.733214 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:17.733477 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 03:50:17.733611 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:50:17.735127 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 28461
I20260810 03:50:17.752662 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 28698
I20260810 03:50:17.793419 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 28938
I20260810 03:50:17.831342 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 28288
2026-08-10T03:50:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2971 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:50:17 dist-test-slave-pww0 krb5kdc[29200](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:50:17 dist-test-slave-pww0 krb5kdc[29200](info): set up 2 sockets
Aug 10 03:50:17 dist-test-slave-pww0 krb5kdc[29200](info): commencing operation
krb5kdc: starting...
W20260810 03:50:19.036154 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:50:19 dist-test-slave-pww0 krb5kdc[29200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:50:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:50:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:50:19.174340 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:37683
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:37683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:19.291689 29225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:19.291981 29225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:19.292023 29225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:19.296118 29225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:50:19.296191 29225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:19.296209 29225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:19.296226 29225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:50:19.296239 29225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:50:19.302034 29225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:37683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:37683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:19.303249 29225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:19.304638 29225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:19.314064 29239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:19.314054 29231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:19.314054 29234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:19.314388 29225 server_base.cc:1034] running on GCE node
I20260810 03:50:19.315026 29225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:19.316293 29225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:19.317485 29225 hybrid_clock.cc:648] HybridClock initialized: now 1786333819317460 us; error 39 us; skew 500 ppm
Aug 10 03:50:19 dist-test-slave-pww0 krb5kdc[29200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333819, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:19.323513 29225 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:50:19.324612 29225 webserver.cc:460] Webserver started at http://127.27.153.126:33081/ using document root <none> and password file <none>
I20260810 03:50:19.325214 29225 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:19.325285 29225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:19.325492 29225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:19.327836 29225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "d115e759037d46c28360d6f43107dbe4"
format_stamp: "Formatted at 2026-08-10 03:50:19 on dist-test-slave-pww0"
server_key: "e1fc12faefeb4bd308d4580c5b709b0f"
server_key_iv: "47363a2cd88fd5ce3fcb5f504308de49"
server_key_version: "encryptionkey@0"
I20260810 03:50:19.328472 29225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "d115e759037d46c28360d6f43107dbe4"
format_stamp: "Formatted at 2026-08-10 03:50:19 on dist-test-slave-pww0"
server_key: "e1fc12faefeb4bd308d4580c5b709b0f"
server_key_iv: "47363a2cd88fd5ce3fcb5f504308de49"
server_key_version: "encryptionkey@0"
I20260810 03:50:19.332619 29225 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:50:19.335438 29253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:19.336894 29225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:19.337025 29225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "d115e759037d46c28360d6f43107dbe4"
format_stamp: "Formatted at 2026-08-10 03:50:19 on dist-test-slave-pww0"
server_key: "e1fc12faefeb4bd308d4580c5b709b0f"
server_key_iv: "47363a2cd88fd5ce3fcb5f504308de49"
server_key_version: "encryptionkey@0"
I20260810 03:50:19.337157 29225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:19.354419 29225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:19.358201 29225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:19.358420 29225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:19.365994 29225 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:37683
I20260810 03:50:19.366011 29360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:37683 every 8 connection(s)
I20260810 03:50:19.367050 29225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:50:19.370199 29361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:19.372234 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 29225
I20260810 03:50:19.372359 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:50:19.372598 28261 external_mini_cluster.cc:1428] Setting key cbd638d0c5c161f922fe7226715ab125
I20260810 03:50:19.377184 29361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d115e759037d46c28360d6f43107dbe4: Bootstrap starting.
I20260810 03:50:19.379988 29361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d115e759037d46c28360d6f43107dbe4: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:19.380990 29361 log.cc:824] T 00000000000000000000000000000000 P d115e759037d46c28360d6f43107dbe4: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:50:19 dist-test-slave-pww0 krb5kdc[29200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:19.383580 29361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d115e759037d46c28360d6f43107dbe4: No bootstrap required, opened a new log
I20260810 03:50:19.387781 29368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:19.374010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50550 (local address 127.27.153.126:37683)
0810 03:50:19.374456 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:19.374473 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:19.374511 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:50:19.375183 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:19.375203 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:19.375602 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:19.375905 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:19.376859 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:19.377898 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:19.379068 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:19.379627 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:19.379760 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:19.382829 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:19.382835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:19.382850 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:19.382880 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:19.384691 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:19.385204 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:19.385209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:19.385214 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:19.385345 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:19.385632 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:19.385634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:19.385635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:19.386055 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:19.386191 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:19.386593 (+ 402us) server_negotiation.cc:299] Negotiation successful
0810 03:50:19.386914 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":131,"threads_started":1}
I20260810 03:50:19.388264 29361 raft_consensus.cc:348] T 00000000000000000000000000000000 P d115e759037d46c28360d6f43107dbe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d115e759037d46c28360d6f43107dbe4" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37683 } }
I20260810 03:50:19.388521 29361 raft_consensus.cc:374] T 00000000000000000000000000000000 P d115e759037d46c28360d6f43107dbe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:19.388581 29361 raft_consensus.cc:729] T 00000000000000000000000000000000 P d115e759037d46c28360d6f43107dbe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d115e759037d46c28360d6f43107dbe4, State: Initialized, Role: FOLLOWER
I20260810 03:50:19.389001 29361 consensus_queue.cc:260] T 00000000000000000000000000000000 P d115e759037d46c28360d6f43107dbe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d115e759037d46c28360d6f43107dbe4" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37683 } }
I20260810 03:50:19.389138 29361 raft_consensus.cc:388] T 00000000000000000000000000000000 P d115e759037d46c28360d6f43107dbe4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:50:19.389205 29361 raft_consensus.cc:482] T 00000000000000000000000000000000 P d115e759037d46c28360d6f43107dbe4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:50:19.389307 29361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d115e759037d46c28360d6f43107dbe4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:19.390286 29361 raft_consensus.cc:504] T 00000000000000000000000000000000 P d115e759037d46c28360d6f43107dbe4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d115e759037d46c28360d6f43107dbe4" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37683 } }
I20260810 03:50:19.390640 29361 leader_election.cc:302] T 00000000000000000000000000000000 P d115e759037d46c28360d6f43107dbe4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d115e759037d46c28360d6f43107dbe4; no voters:
I20260810 03:50:19.390961 29361 leader_election.cc:288] T 00000000000000000000000000000000 P d115e759037d46c28360d6f43107dbe4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:50:19.391074 29371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d115e759037d46c28360d6f43107dbe4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:19.391297 29371 raft_consensus.cc:686] T 00000000000000000000000000000000 P d115e759037d46c28360d6f43107dbe4 [term 1 LEADER]: Becoming Leader. State: Replica: d115e759037d46c28360d6f43107dbe4, State: Running, Role: LEADER
I20260810 03:50:19.391637 29371 consensus_queue.cc:237] T 00000000000000000000000000000000 P d115e759037d46c28360d6f43107dbe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d115e759037d46c28360d6f43107dbe4" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37683 } }
I20260810 03:50:19.392233 29361 sys_catalog.cc:564] T 00000000000000000000000000000000 P d115e759037d46c28360d6f43107dbe4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:50:19.393244 29374 sys_catalog.cc:455] T 00000000000000000000000000000000 P d115e759037d46c28360d6f43107dbe4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d115e759037d46c28360d6f43107dbe4. Latest consensus state: current_term: 1 leader_uuid: "d115e759037d46c28360d6f43107dbe4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d115e759037d46c28360d6f43107dbe4" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37683 } } }
I20260810 03:50:19.393370 29374 sys_catalog.cc:458] T 00000000000000000000000000000000 P d115e759037d46c28360d6f43107dbe4 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:19.393668 29372 sys_catalog.cc:455] T 00000000000000000000000000000000 P d115e759037d46c28360d6f43107dbe4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d115e759037d46c28360d6f43107dbe4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d115e759037d46c28360d6f43107dbe4" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37683 } } }
I20260810 03:50:19.393780 29372 sys_catalog.cc:458] T 00000000000000000000000000000000 P d115e759037d46c28360d6f43107dbe4 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:19.394069 29386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:50:19.397229 29386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:50:19.402997 29386 catalog_manager.cc:1347] Generated new cluster ID: 69c390e29c6a41cfaecdc0a95fb340e3
I20260810 03:50:19.403088 29386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:50:19.416473 29386 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:50:19.417856 29386 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:50:19.423389 29386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d115e759037d46c28360d6f43107dbe4: Generated new TSK 0
I20260810 03:50:19.424314 29386 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.153.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
I20260810 03:50:19.463634 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:37683
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:19.587695 29413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:19.588016 29413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:19.588073 29413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:19.592432 29413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:19.592537 29413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:19.592617 29413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:50:19.598644 29413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:37683
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:19.599987 29413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:19.601151 29413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:19.612001 29425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:19.611994 29424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:19.611994 29429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:19.612372 29413 server_base.cc:1034] running on GCE node
I20260810 03:50:19.612913 29413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:19.613641 29413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:19.614842 29413 hybrid_clock.cc:648] HybridClock initialized: now 1786333819614816 us; error 41 us; skew 500 ppm
Aug 10 03:50:19 dist-test-slave-pww0 krb5kdc[29200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333819, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:19.617914 29413 init.cc:377] Logged in from keytab as kudu/127.27.153.65@KRBTEST.COM (short username kudu)
I20260810 03:50:19.618921 29413 webserver.cc:460] Webserver started at http://127.27.153.65:33095/ using document root <none> and password file <none>
I20260810 03:50:19.619550 29413 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:19.619630 29413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:19.619901 29413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:19.622174 29413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "465f9559a838456291d345b9f9a35ba9"
format_stamp: "Formatted at 2026-08-10 03:50:19 on dist-test-slave-pww0"
server_key: "dbd7ff073f7d27d8631dad8afceed6f9"
server_key_iv: "f31a9efcd67db5628114c7ebedbf6ac2"
server_key_version: "encryptionkey@0"
I20260810 03:50:19.622857 29413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "465f9559a838456291d345b9f9a35ba9"
format_stamp: "Formatted at 2026-08-10 03:50:19 on dist-test-slave-pww0"
server_key: "dbd7ff073f7d27d8631dad8afceed6f9"
server_key_iv: "f31a9efcd67db5628114c7ebedbf6ac2"
server_key_version: "encryptionkey@0"
I20260810 03:50:19.626940 29413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260810 03:50:19.629844 29441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:19.631379 29413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:50:19.631569 29413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "465f9559a838456291d345b9f9a35ba9"
format_stamp: "Formatted at 2026-08-10 03:50:19 on dist-test-slave-pww0"
server_key: "dbd7ff073f7d27d8631dad8afceed6f9"
server_key_iv: "f31a9efcd67db5628114c7ebedbf6ac2"
server_key_version: "encryptionkey@0"
I20260810 03:50:19.631698 29413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:19.652802 29413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:19.656414 29413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:19.656628 29413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:19.657297 29413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:19.658268 29413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:19.658324 29413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:19.658384 29413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:19.658413 29413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:19.667680 29413 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:41965
I20260810 03:50:19.667711 29638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:41965 every 8 connection(s)
I20260810 03:50:19.668669 29413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:50:19.672269 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 29413
I20260810 03:50:19.672390 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:50:19.672672 28261 external_mini_cluster.cc:1428] Setting key f1fdd52d15570df2493787a0d6c4fcd3
Aug 10 03:50:19 dist-test-slave-pww0 krb5kdc[29200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333819, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.153.66@KRBTEST.COM; defaulting to no policy
I20260810 03:50:19.682937 29368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:19.670860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:33241 (local address 127.27.153.126:37683)
0810 03:50:19.670999 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:19.671002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:19.671940 (+ 938us) server_negotiation.cc:407] Connection header received
0810 03:50:19.673027 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:19.673030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:19.673093 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:19.673196 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:19.674470 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:19.675000 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:19.676231 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:19.676463 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:19.676557 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:19.679065 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:19.679068 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:19.679070 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:19.679097 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:19.680710 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:19.681306 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:19.681309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:19.681311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:19.681396 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:19.681735 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:19.681745 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:19.681746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:19.681949 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:19.682038 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:19.682614 (+ 576us) server_negotiation.cc:299] Negotiation successful
0810 03:50:19.682756 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:50:19.683925 29646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:19.671178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:37683 (local address 127.27.153.65:33241)
0810 03:50:19.671748 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:19.671799 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:19.672786 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:19.673405 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:19.673413 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:19.673801 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:19.674319 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:19.674334 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:19.675125 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:19.675127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:19.676101 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:19.676113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:19.676638 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:19.676643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:19.676808 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:19.677403 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:19.677421 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:19.678889 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:19.680880 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:19.680888 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:19.680908 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:19.681196 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:19.681485 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:19.681488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:19.681490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:19.681647 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:19.682054 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:19.682062 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:19.682372 (+ 310us) client_negotiation.cc:769] Sending connection context
0810 03:50:19.682555 (+ 183us) client_negotiation.cc:240] Negotiation successful
0810 03:50:19.682923 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":119,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.66@KRBTEST.COM" created.
I20260810 03:50:19.685307 29642 heartbeater.cc:347] Connected to a master server at 127.27.153.126:37683
I20260810 03:50:19.685559 29642 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:19.686074 29642 heartbeater.cc:511] Master 127.27.153.126:37683 requested a full tablet report, sending...
I20260810 03:50:19.687647 29297 ts_manager.cc:194] Registered new tserver with Master: 465f9559a838456291d345b9f9a35ba9 (127.27.153.65:41965)
I20260810 03:50:19.688803 29297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:33241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
I20260810 03:50:19.702698 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:37683
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:19.820869 29654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:19.821195 29654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:19.821244 29654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:19.825402 29654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:19.825479 29654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:19.825543 29654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:50:19.831148 29654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:37683
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:19.832386 29654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:19.833443 29654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:50:19.844066 29654 server_base.cc:1034] running on GCE node
W20260810 03:50:19.844007 29664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:19.844007 29661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:19.844007 29667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:19.844728 29654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:19.845480 29654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:19.846666 29654 hybrid_clock.cc:648] HybridClock initialized: now 1786333819846643 us; error 38 us; skew 500 ppm
Aug 10 03:50:19 dist-test-slave-pww0 krb5kdc[29200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333819, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:19.849682 29654 init.cc:377] Logged in from keytab as kudu/127.27.153.66@KRBTEST.COM (short username kudu)
I20260810 03:50:19.850770 29654 webserver.cc:460] Webserver started at http://127.27.153.66:34693/ using document root <none> and password file <none>
I20260810 03:50:19.851370 29654 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:19.851441 29654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:19.851691 29654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:19.853892 29654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "b3e23bbcc9be41c891c0618305348c26"
format_stamp: "Formatted at 2026-08-10 03:50:19 on dist-test-slave-pww0"
server_key: "c90b3559380119572c902d11a58d8272"
server_key_iv: "5afdf26958ed93ca7bb8c1c592205673"
server_key_version: "encryptionkey@0"
I20260810 03:50:19.854521 29654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "b3e23bbcc9be41c891c0618305348c26"
format_stamp: "Formatted at 2026-08-10 03:50:19 on dist-test-slave-pww0"
server_key: "c90b3559380119572c902d11a58d8272"
server_key_iv: "5afdf26958ed93ca7bb8c1c592205673"
server_key_version: "encryptionkey@0"
I20260810 03:50:19.858853 29654 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 03:50:19.861725 29680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:19.863147 29654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:50:19.863286 29654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "b3e23bbcc9be41c891c0618305348c26"
format_stamp: "Formatted at 2026-08-10 03:50:19 on dist-test-slave-pww0"
server_key: "c90b3559380119572c902d11a58d8272"
server_key_iv: "5afdf26958ed93ca7bb8c1c592205673"
server_key_version: "encryptionkey@0"
I20260810 03:50:19.863411 29654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:19.893508 29654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:19.896726 29654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:19.896903 29654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:19.897493 29654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:19.898423 29654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:19.898475 29654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:19.898517 29654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:19.898537 29654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:19.907447 29654 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:46773
I20260810 03:50:19.907562 29902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:46773 every 8 connection(s)
I20260810 03:50:19.908460 29654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:50:19.910786 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 29654
I20260810 03:50:19.910902 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:50:19.911185 28261 external_mini_cluster.cc:1428] Setting key e3211f73122b337d06ba073b8fa7a858
Aug 10 03:50:19 dist-test-slave-pww0 krb5kdc[29200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333819, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.153.67@KRBTEST.COM; defaulting to no policy
I20260810 03:50:19.924134 29368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:19.910626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:60027 (local address 127.27.153.126:37683)
0810 03:50:19.910771 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:19.910775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:19.912089 (+ 1314us) server_negotiation.cc:407] Connection header received
0810 03:50:19.913143 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:19.913146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:19.913196 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:19.913279 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:19.914974 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:19.915599 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:19.916861 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:19.917096 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:19.917194 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:19.920177 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:19.920180 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:19.920182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:19.920213 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:19.921745 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:19.922423 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:19.922427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:19.922428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:19.922516 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:19.922881 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:19.922885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:19.922887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:19.923135 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:19.923256 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:19.923885 (+ 629us) server_negotiation.cc:299] Negotiation successful
0810 03:50:19.924005 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.67@KRBTEST.COM" created.
I20260810 03:50:19.925155 29908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:19.911243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:37683 (local address 127.27.153.66:60027)
0810 03:50:19.911934 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:19.911988 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:19.912904 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:19.913497 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:19.913507 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:19.914039 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:19.914767 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:19.914789 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:19.915745 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:19.915749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:19.916720 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:19.916731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:19.917269 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:19.917274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:19.917465 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:19.918111 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:19.918138 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:19.919981 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:19.921932 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:19.921940 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:19.921958 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:19.922304 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:19.922608 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:19.922611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:19.922613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:19.922786 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:19.923276 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:19.923282 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:19.923632 (+ 350us) client_negotiation.cc:769] Sending connection context
0810 03:50:19.923823 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 03:50:19.924184 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":213,"threads_started":1}
I20260810 03:50:19.926402 29904 heartbeater.cc:347] Connected to a master server at 127.27.153.126:37683
I20260810 03:50:19.926699 29904 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:19.927258 29904 heartbeater.cc:511] Master 127.27.153.126:37683 requested a full tablet report, sending...
I20260810 03:50:19.928465 29297 ts_manager.cc:194] Registered new tserver with Master: b3e23bbcc9be41c891c0618305348c26 (127.27.153.66:46773)
I20260810 03:50:19.929193 29297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:60027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
I20260810 03:50:19.940903 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:37683
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:20.057097 29917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:20.057410 29917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:20.057457 29917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:20.061606 29917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:20.061688 29917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:20.061753 29917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:50:20.067257 29917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:37683
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:20.068544 29917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:20.069672 29917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:20.079607 29922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:20.079605 29928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:20.079669 29923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:20.079937 29917 server_base.cc:1034] running on GCE node
I20260810 03:50:20.080478 29917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:20.081180 29917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:20.082345 29917 hybrid_clock.cc:648] HybridClock initialized: now 1786333820082325 us; error 33 us; skew 500 ppm
Aug 10 03:50:20 dist-test-slave-pww0 krb5kdc[29200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333820, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:20.085212 29917 init.cc:377] Logged in from keytab as kudu/127.27.153.67@KRBTEST.COM (short username kudu)
I20260810 03:50:20.086138 29917 webserver.cc:460] Webserver started at http://127.27.153.67:44201/ using document root <none> and password file <none>
I20260810 03:50:20.086697 29917 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:20.086758 29917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:20.086932 29917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:20.089104 29917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "e16ba71450dd46e58912756f31b3fc11"
format_stamp: "Formatted at 2026-08-10 03:50:20 on dist-test-slave-pww0"
server_key: "2c4c1b76d0ea329737de85adc75b13f1"
server_key_iv: "6282a6c487350d9367bd538736b7aa75"
server_key_version: "encryptionkey@0"
I20260810 03:50:20.089794 29917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "e16ba71450dd46e58912756f31b3fc11"
format_stamp: "Formatted at 2026-08-10 03:50:20 on dist-test-slave-pww0"
server_key: "2c4c1b76d0ea329737de85adc75b13f1"
server_key_iv: "6282a6c487350d9367bd538736b7aa75"
server_key_version: "encryptionkey@0"
I20260810 03:50:20.094240 29917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:50:20.096895 29940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:20.098194 29917 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 03:50:20.098331 29917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "e16ba71450dd46e58912756f31b3fc11"
format_stamp: "Formatted at 2026-08-10 03:50:20 on dist-test-slave-pww0"
server_key: "2c4c1b76d0ea329737de85adc75b13f1"
server_key_iv: "6282a6c487350d9367bd538736b7aa75"
server_key_version: "encryptionkey@0"
I20260810 03:50:20.098449 29917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:20.116439 29917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:20.119725 29917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:20.119927 29917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:20.120565 29917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:20.121443 29917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:20.121491 29917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:20.121549 29917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:20.121577 29917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:20.130784 29917 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:39819
I20260810 03:50:20.130873 30156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:39819 every 8 connection(s)
I20260810 03:50:20.131798 29917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:50:20.138890 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 29917
I20260810 03:50:20.139014 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:50:20.139297 28261 external_mini_cluster.cc:1428] Setting key 0666315cfac018bd1df4af87ed7139db
Aug 10 03:50:20 dist-test-slave-pww0 krb5kdc[29200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333820, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:20.145260 29368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:20.133655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:56355 (local address 127.27.153.126:37683)
0810 03:50:20.133879 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:20.133883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:20.134651 (+ 768us) server_negotiation.cc:407] Connection header received
0810 03:50:20.135671 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:20.135674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:20.135736 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:20.135808 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:20.137080 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.137620 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.138874 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.139149 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.139245 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:20.141984 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:20.141987 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:20.141989 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:20.142016 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:20.143415 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:20.143946 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:20.143949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:20.143950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:20.144026 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:20.144299 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:20.144301 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:20.144302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:20.144479 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:20.144549 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:20.145035 (+ 486us) server_negotiation.cc:299] Negotiation successful
0810 03:50:20.145138 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260810 03:50:20.146174 30160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:20.134030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:37683 (local address 127.27.153.67:56355)
0810 03:50:20.134495 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:20.134538 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:20.135425 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:20.136043 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:20.136050 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:20.136420 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:20.136926 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:20.136941 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.137746 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.137749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:20.138721 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:20.138734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.139324 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.139330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:20.139580 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:20.140183 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:20.140202 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:20.141826 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:20.143555 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:20.143561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:20.143575 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:20.143860 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:20.144091 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:20.144093 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:20.144094 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:20.144231 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:20.144550 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:20.144556 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:20.144840 (+ 284us) client_negotiation.cc:769] Sending connection context
0810 03:50:20.145007 (+ 167us) client_negotiation.cc:240] Negotiation successful
0810 03:50:20.145284 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":104,"threads_started":1}
I20260810 03:50:20.147332 30157 heartbeater.cc:347] Connected to a master server at 127.27.153.126:37683
I20260810 03:50:20.147600 30157 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:20.148128 30157 heartbeater.cc:511] Master 127.27.153.126:37683 requested a full tablet report, sending...
I20260810 03:50:20.149200 29297 ts_manager.cc:194] Registered new tserver with Master: e16ba71450dd46e58912756f31b3fc11 (127.27.153.67:39819)
I20260810 03:50:20.149783 29297 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:56355
I20260810 03:50:20.153340 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:50:20.165643 29368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:20.157945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50552 (local address 127.27.153.126:37683)
0810 03:50:20.158090 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:20.158095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:20.158290 (+ 195us) server_negotiation.cc:407] Connection header received
0810 03:50:20.158447 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:20.158449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:20.158496 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:20.158568 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:20.159330 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.159923 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.160762 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.161011 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.161106 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:20.162517 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:20.162520 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:20.162523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:20.162552 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:20.163952 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:20.164385 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:20.164388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:20.164389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:20.164473 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:20.164730 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:20.164733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:20.164735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:20.164982 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:20.165107 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:20.165360 (+ 253us) server_negotiation.cc:299] Negotiation successful
0810 03:50:20.165481 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 03:50:20.172283 29297 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:50:20.175765 29297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:50:20.194937 30180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:20.186947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:41965 (local address 127.0.0.1:35730)
0810 03:50:20.188684 (+ 1737us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:20.188745 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:20.188895 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:20.189665 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:20.189673 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:20.189716 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:20.189862 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:20.189869 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.191816 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.191819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:20.193201 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:20.193213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.194340 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.194344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:20.194482 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:20.194529 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 03:50:20.194644 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 03:50:20.194768 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1536,"mutex_wait_us":2,"thread_start_us":99,"threads_started":1}
I20260810 03:50:20.195746 30181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:20.187819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35730 (local address 127.27.153.65:41965)
0810 03:50:20.189112 (+ 1293us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:20.189120 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:20.189146 (+ 26us) server_negotiation.cc:407] Connection header received
0810 03:50:20.189223 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:20.189228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:20.189449 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:20.189636 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:20.190490 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.191691 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.193334 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.194220 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.194339 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:20.194582 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:20.195447 (+ 865us) server_negotiation.cc:299] Negotiation successful
0810 03:50:20.195594 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1181,"thread_start_us":121,"threads_started":1}
I20260810 03:50:20.197088 30186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:20.188892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47784 (local address 127.27.153.67:39819)
0810 03:50:20.189273 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:20.189294 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:20.191843 (+ 2549us) server_negotiation.cc:407] Connection header received
0810 03:50:20.191929 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:20.191935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:20.192139 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:20.192301 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:20.192677 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.193852 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.195512 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.196345 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.196416 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:20.196522 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:20.196830 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 03:50:20.196936 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":175,"threads_started":1}
I20260810 03:50:20.197525 30184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:20.188625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:39819 (local address 127.0.0.1:47784)
0810 03:50:20.190246 (+ 1621us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:20.190260 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:20.190373 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:20.192294 (+ 1921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:20.192297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:20.192346 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:20.192480 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:20.192489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.194010 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.194014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:20.195318 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:20.195330 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.196604 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.196606 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:20.196674 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:20.196690 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 03:50:20.197299 (+ 609us) client_negotiation.cc:240] Negotiation successful
0810 03:50:20.197392 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1565,"mutex_wait_us":21,"thread_start_us":154,"threads_started":1}
I20260810 03:50:20.200016 30187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:20.189946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:46773 (local address 127.0.0.1:43020)
0810 03:50:20.192604 (+ 2658us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:20.192619 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:20.192711 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:20.194647 (+ 1936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:20.194650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:20.194671 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:20.194764 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:20.194770 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.197153 (+ 2383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.197157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:20.198305 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:20.198318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.199616 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.199620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:20.199714 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:20.199737 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 03:50:20.199802 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 03:50:20.199875 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2594,"thread_start_us":106,"threads_started":1}
I20260810 03:50:20.199759 29526 tablet_service.cc:1467] Processing CreateTablet for tablet 500cd33bf92f4f7d86d60ed50f3e36d1 (DEFAULT_TABLE table=test-table [id=fd9953dae4d3470b88ce9126e1552818]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:20.200574 30190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:20.190190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43020 (local address 127.27.153.66:46773)
0810 03:50:20.190636 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:20.190647 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:20.192806 (+ 2159us) server_negotiation.cc:407] Connection header received
0810 03:50:20.192890 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:20.192894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:20.193081 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:20.193215 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:20.194993 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.196993 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.198460 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.199277 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.199391 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:20.199552 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:20.200306 (+ 754us) server_negotiation.cc:299] Negotiation successful
0810 03:50:20.200433 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":294,"threads_started":1}
I20260810 03:50:20.200719 30029 tablet_service.cc:1467] Processing CreateTablet for tablet 500cd33bf92f4f7d86d60ed50f3e36d1 (DEFAULT_TABLE table=test-table [id=fd9953dae4d3470b88ce9126e1552818]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:20.201220 29526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500cd33bf92f4f7d86d60ed50f3e36d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:20.201854 30029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500cd33bf92f4f7d86d60ed50f3e36d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:20.203321 29766 tablet_service.cc:1467] Processing CreateTablet for tablet 500cd33bf92f4f7d86d60ed50f3e36d1 (DEFAULT_TABLE table=test-table [id=fd9953dae4d3470b88ce9126e1552818]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:20.204347 29766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 500cd33bf92f4f7d86d60ed50f3e36d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:20.209473 30194 tablet_bootstrap.cc:492] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9: Bootstrap starting.
I20260810 03:50:20.211915 30196 tablet_bootstrap.cc:492] T 500cd33bf92f4f7d86d60ed50f3e36d1 P e16ba71450dd46e58912756f31b3fc11: Bootstrap starting.
I20260810 03:50:20.212847 30194 tablet_bootstrap.cc:654] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:20.213830 30194 log.cc:824] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:20.214401 30196 tablet_bootstrap.cc:654] T 500cd33bf92f4f7d86d60ed50f3e36d1 P e16ba71450dd46e58912756f31b3fc11: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:20.214764 30195 tablet_bootstrap.cc:492] T 500cd33bf92f4f7d86d60ed50f3e36d1 P b3e23bbcc9be41c891c0618305348c26: Bootstrap starting.
I20260810 03:50:20.216073 30196 log.cc:824] T 500cd33bf92f4f7d86d60ed50f3e36d1 P e16ba71450dd46e58912756f31b3fc11: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:20.217197 30195 tablet_bootstrap.cc:654] T 500cd33bf92f4f7d86d60ed50f3e36d1 P b3e23bbcc9be41c891c0618305348c26: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:20.217386 30194 tablet_bootstrap.cc:492] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9: No bootstrap required, opened a new log
I20260810 03:50:20.217595 30194 ts_tablet_manager.cc:1397] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 03:50:20.218357 30196 tablet_bootstrap.cc:492] T 500cd33bf92f4f7d86d60ed50f3e36d1 P e16ba71450dd46e58912756f31b3fc11: No bootstrap required, opened a new log
I20260810 03:50:20.218458 30195 log.cc:824] T 500cd33bf92f4f7d86d60ed50f3e36d1 P b3e23bbcc9be41c891c0618305348c26: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:20.218567 30196 ts_tablet_manager.cc:1397] T 500cd33bf92f4f7d86d60ed50f3e36d1 P e16ba71450dd46e58912756f31b3fc11: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 03:50:20.220865 30195 tablet_bootstrap.cc:492] T 500cd33bf92f4f7d86d60ed50f3e36d1 P b3e23bbcc9be41c891c0618305348c26: No bootstrap required, opened a new log
I20260810 03:50:20.221089 30195 ts_tablet_manager.cc:1397] T 500cd33bf92f4f7d86d60ed50f3e36d1 P b3e23bbcc9be41c891c0618305348c26: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 03:50:20.221534 30194 raft_consensus.cc:348] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e23bbcc9be41c891c0618305348c26" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 46773 } } peers { permanent_uuid: "465f9559a838456291d345b9f9a35ba9" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 41965 } } peers { permanent_uuid: "e16ba71450dd46e58912756f31b3fc11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 39819 } }
I20260810 03:50:20.221791 30194 raft_consensus.cc:374] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:20.221858 30194 raft_consensus.cc:729] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 465f9559a838456291d345b9f9a35ba9, State: Initialized, Role: FOLLOWER
I20260810 03:50:20.222304 30194 consensus_queue.cc:260] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e23bbcc9be41c891c0618305348c26" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 46773 } } peers { permanent_uuid: "465f9559a838456291d345b9f9a35ba9" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 41965 } } peers { permanent_uuid: "e16ba71450dd46e58912756f31b3fc11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 39819 } }
I20260810 03:50:20.222344 30196 raft_consensus.cc:348] T 500cd33bf92f4f7d86d60ed50f3e36d1 P e16ba71450dd46e58912756f31b3fc11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e23bbcc9be41c891c0618305348c26" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 46773 } } peers { permanent_uuid: "465f9559a838456291d345b9f9a35ba9" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 41965 } } peers { permanent_uuid: "e16ba71450dd46e58912756f31b3fc11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 39819 } }
I20260810 03:50:20.222622 30196 raft_consensus.cc:374] T 500cd33bf92f4f7d86d60ed50f3e36d1 P e16ba71450dd46e58912756f31b3fc11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:20.222674 30196 raft_consensus.cc:729] T 500cd33bf92f4f7d86d60ed50f3e36d1 P e16ba71450dd46e58912756f31b3fc11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e16ba71450dd46e58912756f31b3fc11, State: Initialized, Role: FOLLOWER
I20260810 03:50:20.223116 30196 consensus_queue.cc:260] T 500cd33bf92f4f7d86d60ed50f3e36d1 P e16ba71450dd46e58912756f31b3fc11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e23bbcc9be41c891c0618305348c26" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 46773 } } peers { permanent_uuid: "465f9559a838456291d345b9f9a35ba9" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 41965 } } peers { permanent_uuid: "e16ba71450dd46e58912756f31b3fc11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 39819 } }
I20260810 03:50:20.223313 30194 ts_tablet_manager.cc:1428] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 03:50:20.223905 29642 heartbeater.cc:503] Master 127.27.153.126:37683 was elected leader, sending a full tablet report...
I20260810 03:50:20.224205 30196 ts_tablet_manager.cc:1428] T 500cd33bf92f4f7d86d60ed50f3e36d1 P e16ba71450dd46e58912756f31b3fc11: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260810 03:50:20.224572 30157 heartbeater.cc:503] Master 127.27.153.126:37683 was elected leader, sending a full tablet report...
I20260810 03:50:20.225068 30195 raft_consensus.cc:348] T 500cd33bf92f4f7d86d60ed50f3e36d1 P b3e23bbcc9be41c891c0618305348c26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e23bbcc9be41c891c0618305348c26" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 46773 } } peers { permanent_uuid: "465f9559a838456291d345b9f9a35ba9" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 41965 } } peers { permanent_uuid: "e16ba71450dd46e58912756f31b3fc11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 39819 } }
I20260810 03:50:20.225368 30195 raft_consensus.cc:374] T 500cd33bf92f4f7d86d60ed50f3e36d1 P b3e23bbcc9be41c891c0618305348c26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:20.225419 30195 raft_consensus.cc:729] T 500cd33bf92f4f7d86d60ed50f3e36d1 P b3e23bbcc9be41c891c0618305348c26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e23bbcc9be41c891c0618305348c26, State: Initialized, Role: FOLLOWER
I20260810 03:50:20.225906 30195 consensus_queue.cc:260] T 500cd33bf92f4f7d86d60ed50f3e36d1 P b3e23bbcc9be41c891c0618305348c26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e23bbcc9be41c891c0618305348c26" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 46773 } } peers { permanent_uuid: "465f9559a838456291d345b9f9a35ba9" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 41965 } } peers { permanent_uuid: "e16ba71450dd46e58912756f31b3fc11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 39819 } }
I20260810 03:50:20.226553 29904 heartbeater.cc:503] Master 127.27.153.126:37683 was elected leader, sending a full tablet report...
I20260810 03:50:20.226925 30195 ts_tablet_manager.cc:1428] T 500cd33bf92f4f7d86d60ed50f3e36d1 P b3e23bbcc9be41c891c0618305348c26: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 03:50:20.313983 30200 raft_consensus.cc:482] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:50:20.314204 30200 raft_consensus.cc:504] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e23bbcc9be41c891c0618305348c26" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 46773 } } peers { permanent_uuid: "465f9559a838456291d345b9f9a35ba9" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 41965 } } peers { permanent_uuid: "e16ba71450dd46e58912756f31b3fc11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 39819 } }
I20260810 03:50:20.315493 30200 leader_election.cc:288] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3e23bbcc9be41c891c0618305348c26 (127.27.153.66:46773), e16ba71450dd46e58912756f31b3fc11 (127.27.153.67:39819)
I20260810 03:50:20.319973 30186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:20.316014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:57261 (local address 127.27.153.67:39819)
0810 03:50:20.316157 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:20.316173 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:20.316301 (+ 128us) server_negotiation.cc:407] Connection header received
0810 03:50:20.316557 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:20.316561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:20.316618 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:20.316696 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:20.317069 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.317690 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.318896 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.319331 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.319431 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:20.319513 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:20.319743 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 03:50:20.319832 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 03:50:20.319972 30190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:20.315928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:48667 (local address 127.27.153.66:46773)
0810 03:50:20.316063 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:20.316068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:20.316604 (+ 536us) server_negotiation.cc:407] Connection header received
0810 03:50:20.316857 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:20.316861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:20.316918 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:20.317008 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:20.317354 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.317931 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.318966 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.319399 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.319541 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:20.319595 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:20.319750 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 03:50:20.319832 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:50:20.320000 30205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:20.315776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:39819 (local address 127.27.153.65:57261)
0810 03:50:20.316189 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:20.316205 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:20.316386 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:20.316735 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:20.316738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:20.316782 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:20.316930 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:20.316937 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.317831 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.317834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:20.318758 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:20.318769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.319481 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.319486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:20.319582 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:20.319607 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 03:50:20.319702 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 03:50:20.319832 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":19,"thread_start_us":128,"threads_started":1}
I20260810 03:50:20.320009 30206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:20.315779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:46773 (local address 127.27.153.65:48667)
0810 03:50:20.316434 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:20.316449 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:20.316680 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:20.317062 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:20.317065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:20.317086 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:20.317204 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:20.317211 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.318041 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.318044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:20.318834 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:20.318846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.319540 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.319544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:20.319629 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:20.319647 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 03:50:20.319724 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 03:50:20.319862 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":337,"thread_start_us":116,"threads_started":1}
I20260810 03:50:20.320894 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500cd33bf92f4f7d86d60ed50f3e36d1" candidate_uuid: "465f9559a838456291d345b9f9a35ba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e23bbcc9be41c891c0618305348c26" is_pre_election: true
I20260810 03:50:20.320894 30069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500cd33bf92f4f7d86d60ed50f3e36d1" candidate_uuid: "465f9559a838456291d345b9f9a35ba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16ba71450dd46e58912756f31b3fc11" is_pre_election: true
I20260810 03:50:20.321313 30069 raft_consensus.cc:2454] T 500cd33bf92f4f7d86d60ed50f3e36d1 P e16ba71450dd46e58912756f31b3fc11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 465f9559a838456291d345b9f9a35ba9 in term 0.
I20260810 03:50:20.321934 29446 leader_election.cc:302] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 465f9559a838456291d345b9f9a35ba9, e16ba71450dd46e58912756f31b3fc11; no voters:
I20260810 03:50:20.322221 30200 raft_consensus.cc:2781] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:50:20.322306 30200 raft_consensus.cc:482] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:50:20.322338 30200 raft_consensus.cc:3037] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:20.323355 29809 raft_consensus.cc:2454] T 500cd33bf92f4f7d86d60ed50f3e36d1 P b3e23bbcc9be41c891c0618305348c26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 465f9559a838456291d345b9f9a35ba9 in term 0.
I20260810 03:50:20.323422 30200 raft_consensus.cc:504] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e23bbcc9be41c891c0618305348c26" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 46773 } } peers { permanent_uuid: "465f9559a838456291d345b9f9a35ba9" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 41965 } } peers { permanent_uuid: "e16ba71450dd46e58912756f31b3fc11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 39819 } }
I20260810 03:50:20.323887 30200 leader_election.cc:288] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9 [CANDIDATE]: Term 1 election: Requested vote from peers b3e23bbcc9be41c891c0618305348c26 (127.27.153.66:46773), e16ba71450dd46e58912756f31b3fc11 (127.27.153.67:39819)
I20260810 03:50:20.324328 30069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500cd33bf92f4f7d86d60ed50f3e36d1" candidate_uuid: "465f9559a838456291d345b9f9a35ba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e16ba71450dd46e58912756f31b3fc11"
I20260810 03:50:20.324424 29809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "500cd33bf92f4f7d86d60ed50f3e36d1" candidate_uuid: "465f9559a838456291d345b9f9a35ba9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e23bbcc9be41c891c0618305348c26"
I20260810 03:50:20.324458 30069 raft_consensus.cc:3037] T 500cd33bf92f4f7d86d60ed50f3e36d1 P e16ba71450dd46e58912756f31b3fc11 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:20.324558 29809 raft_consensus.cc:3037] T 500cd33bf92f4f7d86d60ed50f3e36d1 P b3e23bbcc9be41c891c0618305348c26 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:20.325565 30069 raft_consensus.cc:2454] T 500cd33bf92f4f7d86d60ed50f3e36d1 P e16ba71450dd46e58912756f31b3fc11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 465f9559a838456291d345b9f9a35ba9 in term 1.
I20260810 03:50:20.325680 29809 raft_consensus.cc:2454] T 500cd33bf92f4f7d86d60ed50f3e36d1 P b3e23bbcc9be41c891c0618305348c26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 465f9559a838456291d345b9f9a35ba9 in term 1.
I20260810 03:50:20.325935 29446 leader_election.cc:302] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 465f9559a838456291d345b9f9a35ba9, e16ba71450dd46e58912756f31b3fc11; no voters:
I20260810 03:50:20.326121 30200 raft_consensus.cc:2781] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:20.326368 30200 raft_consensus.cc:686] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9 [term 1 LEADER]: Becoming Leader. State: Replica: 465f9559a838456291d345b9f9a35ba9, State: Running, Role: LEADER
I20260810 03:50:20.326750 30200 consensus_queue.cc:237] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e23bbcc9be41c891c0618305348c26" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 46773 } } peers { permanent_uuid: "465f9559a838456291d345b9f9a35ba9" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 41965 } } peers { permanent_uuid: "e16ba71450dd46e58912756f31b3fc11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 39819 } }
I20260810 03:50:20.330503 29291 catalog_manager.cc:5495] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 465f9559a838456291d345b9f9a35ba9 (127.27.153.65). New cstate: current_term: 1 leader_uuid: "465f9559a838456291d345b9f9a35ba9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e23bbcc9be41c891c0618305348c26" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 46773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "465f9559a838456291d345b9f9a35ba9" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 41965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e16ba71450dd46e58912756f31b3fc11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 39819 } health_report { overall_health: UNKNOWN } } }
Aug 10 03:50:20 dist-test-slave-pww0 krb5kdc[29200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.65@KRBTEST.COM
I20260810 03:50:20.358660 30181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:20.348166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35734 (local address 127.27.153.65:41965)
0810 03:50:20.348327 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:20.348332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:20.348492 (+ 160us) server_negotiation.cc:407] Connection header received
0810 03:50:20.348630 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:20.348634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:20.348683 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:20.348761 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:20.349473 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.349996 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.350876 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.351099 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.351189 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:20.353939 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:20.353958 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:20.353975 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:20.354011 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:20.356369 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:20.356977 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:20.356984 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:20.356985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:20.357098 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:20.357490 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:20.357494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:20.357495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:20.357810 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:20.358017 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:20.358312 (+ 295us) server_negotiation.cc:299] Negotiation successful
0810 03:50:20.358461 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 03:50:20 dist-test-slave-pww0 krb5kdc[29200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333820, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260810 03:50:20.382861 30158 tablet.cc:2369] T 500cd33bf92f4f7d86d60ed50f3e36d1 P e16ba71450dd46e58912756f31b3fc11: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 03:50:20 dist-test-slave-pww0 krb5kdc[29200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333820, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.153.65@KRBTEST.COM
I20260810 03:50:20.390884 30181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:20.381528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35744 (local address 127.27.153.65:41965)
0810 03:50:20.381721 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:20.381727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:20.381869 (+ 142us) server_negotiation.cc:407] Connection header received
0810 03:50:20.382023 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:20.382027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:20.382075 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:20.382152 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:20.382941 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.383589 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.384439 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.384692 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.384789 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:20.387598 (+ 2809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:20.387603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:20.387605 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:20.387640 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:20.389176 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:20.389659 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:20.389661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:20.389663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:20.389770 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:20.389993 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:20.389995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:20.389996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:20.390169 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:20.390305 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:20.390621 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 03:50:20.390756 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 03:50:20 dist-test-slave-pww0 krb5kdc[29200](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260810 03:50:20.409760 29907 tablet.cc:2369] T 500cd33bf92f4f7d86d60ed50f3e36d1 P b3e23bbcc9be41c891c0618305348c26: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 03:50:20 dist-test-slave-pww0 krb5kdc[29200](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.65@KRBTEST.COM
W20260810 03:50:20.420137 29645 tablet.cc:2369] T 500cd33bf92f4f7d86d60ed50f3e36d1 P 465f9559a838456291d345b9f9a35ba9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:50:20.420773 30181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:20.411392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35752 (local address 127.27.153.65:41965)
0810 03:50:20.411553 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:20.411557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:20.411719 (+ 162us) server_negotiation.cc:407] Connection header received
0810 03:50:20.411889 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:20.411891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:20.411939 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:20.412009 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:20.412733 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.413236 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.414095 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:20.414325 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:20.414419 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:20.417179 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:20.417183 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:20.417185 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:20.417219 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:20.418869 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:20.419296 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:20.419299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:20.419300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:20.419379 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:20.419777 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:20.419782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:20.419783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:20.420036 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:20.420244 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:20.420475 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 03:50:20.420614 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 03:50:20.422516 29489 tablet_service.cc:2908] Rejecting scan request for tablet 500cd33bf92f4f7d86d60ed50f3e36d1: Uninitialized: clean time has not yet been initialized
I20260810 03:50:20.424985 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 29413
I20260810 03:50:20.449661 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 29654
I20260810 03:50:20.458689 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 29917
I20260810 03:50:20.477248 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 29225
2026-08-10T03:50:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2644 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:50:20 dist-test-slave-pww0 krb5kdc[30229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:50:20 dist-test-slave-pww0 krb5kdc[30229](info): set up 2 sockets
Aug 10 03:50:20 dist-test-slave-pww0 krb5kdc[30229](info): commencing operation
krb5kdc: starting...
W20260810 03:50:21.687790 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.179s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:50:21 dist-test-slave-pww0 krb5kdc[30229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:50:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:50:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:50:21.816776 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:46701
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:46701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:21.935333 30249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:21.935680 30249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:21.935743 30249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:21.939914 30249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:50:21.939996 30249 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 03:50:21.940021 30249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:21.940042 30249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:21.940065 30249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:50:21.940095 30249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:50:21.945930 30249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:46701
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:46701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:21.947124 30249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:21.948230 30249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:21.957453 30260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:21.957406 30266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:21.957456 30262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:21.957652 30249 server_base.cc:1034] running on GCE node
I20260810 03:50:21.958308 30249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:21.959731 30249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:21.960914 30249 hybrid_clock.cc:648] HybridClock initialized: now 1786333821960899 us; error 36 us; skew 500 ppm
Aug 10 03:50:21 dist-test-slave-pww0 krb5kdc[30229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333821, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:21.963945 30249 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:50:21.964962 30249 webserver.cc:460] Webserver started at http://127.27.153.126:43875/ using document root <none> and password file <none>
I20260810 03:50:21.965524 30249 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:21.965588 30249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:21.965776 30249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:21.968009 30249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "9e14520d9b09461d9e8faa30a03c8af0"
format_stamp: "Formatted at 2026-08-10 03:50:21 on dist-test-slave-pww0"
server_key: "38d747bac6474c011797fafe74995201"
server_key_iv: "e86d8b78078f5bc921fa2e7c05bb5d34"
server_key_version: "encryptionkey@0"
I20260810 03:50:21.968568 30249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "9e14520d9b09461d9e8faa30a03c8af0"
format_stamp: "Formatted at 2026-08-10 03:50:21 on dist-test-slave-pww0"
server_key: "38d747bac6474c011797fafe74995201"
server_key_iv: "e86d8b78078f5bc921fa2e7c05bb5d34"
server_key_version: "encryptionkey@0"
I20260810 03:50:21.972589 30249 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:50:21.975162 30277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:21.976576 30249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 03:50:21.976707 30249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "9e14520d9b09461d9e8faa30a03c8af0"
format_stamp: "Formatted at 2026-08-10 03:50:21 on dist-test-slave-pww0"
server_key: "38d747bac6474c011797fafe74995201"
server_key_iv: "e86d8b78078f5bc921fa2e7c05bb5d34"
server_key_version: "encryptionkey@0"
I20260810 03:50:21.976835 30249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:21.986716 30249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:21.990249 30249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:21.990465 30249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:21.999281 30249 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:46701
I20260810 03:50:21.999336 30387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:46701 every 8 connection(s)
I20260810 03:50:22.000360 30249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:50:22.003340 30388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:22.004637 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 30249
I20260810 03:50:22.004750 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:50:22.004999 28261 external_mini_cluster.cc:1428] Setting key 12fd6d90ec6d662b3dbdd0d45eb3782b
I20260810 03:50:22.010615 30388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0: Bootstrap starting.
I20260810 03:50:22.013407 30388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:22.014200 30388 log.cc:824] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:50:22 dist-test-slave-pww0 krb5kdc[30229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:22.016525 30388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0: No bootstrap required, opened a new log
I20260810 03:50:22.019989 30388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e14520d9b09461d9e8faa30a03c8af0" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 46701 } }
I20260810 03:50:22.020314 30388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:22.020392 30388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e14520d9b09461d9e8faa30a03c8af0, State: Initialized, Role: FOLLOWER
I20260810 03:50:22.020939 30388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e14520d9b09461d9e8faa30a03c8af0" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 46701 } }
I20260810 03:50:22.021100 30388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:50:22.021183 30388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:50:22.021287 30388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:22.021835 30391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:22.006529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56038 (local address 127.27.153.126:46701)
0810 03:50:22.007026 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:22.007046 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:22.007127 (+ 81us) server_negotiation.cc:407] Connection header received
0810 03:50:22.007998 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:22.008024 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:22.008487 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:22.008885 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:22.009596 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.011039 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.012012 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.012709 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.012881 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:22.015666 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:22.015673 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:22.015690 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:22.015721 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:22.018414 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:22.019030 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:22.019036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:22.019042 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:22.019179 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:22.019543 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:22.019546 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:22.019548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:22.020010 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:22.020179 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:22.020591 (+ 412us) server_negotiation.cc:299] Negotiation successful
0810 03:50:22.020902 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":146,"threads_started":1}
I20260810 03:50:22.022312 30388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e14520d9b09461d9e8faa30a03c8af0" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 46701 } }
I20260810 03:50:22.022650 30388 leader_election.cc:302] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e14520d9b09461d9e8faa30a03c8af0; no voters:
I20260810 03:50:22.022997 30388 leader_election.cc:288] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:50:22.023103 30393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:22.023281 30393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0 [term 1 LEADER]: Becoming Leader. State: Replica: 9e14520d9b09461d9e8faa30a03c8af0, State: Running, Role: LEADER
I20260810 03:50:22.023701 30393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e14520d9b09461d9e8faa30a03c8af0" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 46701 } }
I20260810 03:50:22.024461 30388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:50:22.025653 30399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e14520d9b09461d9e8faa30a03c8af0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e14520d9b09461d9e8faa30a03c8af0" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 46701 } } }
I20260810 03:50:22.025804 30399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:22.026063 30398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e14520d9b09461d9e8faa30a03c8af0. Latest consensus state: current_term: 1 leader_uuid: "9e14520d9b09461d9e8faa30a03c8af0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e14520d9b09461d9e8faa30a03c8af0" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 46701 } } }
I20260810 03:50:22.026157 30398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:22.026355 30410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:50:22.029901 30410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:50:22.036593 30410 catalog_manager.cc:1347] Generated new cluster ID: 7a2722d55fce416c9ce15600649fcece
I20260810 03:50:22.036698 30410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:50:22.052520 30410 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:50:22.053534 30410 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:50:22.058846 30410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0: Generated new TSK 0
I20260810 03:50:22.059653 30410 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.153.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
I20260810 03:50:22.099900 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:46701
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:22.250576 30435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:22.250885 30435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:22.250931 30435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:22.255059 30435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:22.255146 30435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:22.255187 30435 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:50:22.255219 30435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:50:22.260713 30435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.153.126:46701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:22.261833 30435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:22.262882 30435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:22.272912 30441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:22.272985 30435 server_base.cc:1034] running on GCE node
W20260810 03:50:22.272912 30442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:22.272926 30447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:22.273589 30435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:22.274307 30435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:22.275564 30435 hybrid_clock.cc:648] HybridClock initialized: now 1786333822275546 us; error 31 us; skew 500 ppm
Aug 10 03:50:22 dist-test-slave-pww0 krb5kdc[30229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333822, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:22.278568 30435 init.cc:377] Logged in from keytab as kudu/127.27.153.65@KRBTEST.COM (short username kudu)
I20260810 03:50:22.279656 30435 webserver.cc:460] Webserver started at http://127.27.153.65:39945/ using document root <none> and password file <none>
I20260810 03:50:22.280253 30435 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:22.280330 30435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:22.280562 30435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:22.282850 30435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "1c2d8d5df0a74479859b89f4fb899221"
format_stamp: "Formatted at 2026-08-10 03:50:22 on dist-test-slave-pww0"
server_key: "a5d9613478339dd944e366bebd5f813a"
server_key_iv: "9e14c29b31cb833a5eefd552045b1ec8"
server_key_version: "encryptionkey@0"
I20260810 03:50:22.283447 30435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "1c2d8d5df0a74479859b89f4fb899221"
format_stamp: "Formatted at 2026-08-10 03:50:22 on dist-test-slave-pww0"
server_key: "a5d9613478339dd944e366bebd5f813a"
server_key_iv: "9e14c29b31cb833a5eefd552045b1ec8"
server_key_version: "encryptionkey@0"
I20260810 03:50:22.287859 30435 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:50:22.290688 30459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:22.292168 30435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:22.292300 30435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "1c2d8d5df0a74479859b89f4fb899221"
format_stamp: "Formatted at 2026-08-10 03:50:22 on dist-test-slave-pww0"
server_key: "a5d9613478339dd944e366bebd5f813a"
server_key_iv: "9e14c29b31cb833a5eefd552045b1ec8"
server_key_version: "encryptionkey@0"
I20260810 03:50:22.292421 30435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:22.324141 30435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:22.327673 30435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:22.327874 30435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:22.329473 30435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:22.329531 30435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:22.329588 30435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:22.329617 30435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:50:22 dist-test-slave-pww0 krb5kdc[30229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333822, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:22.340713 30435 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:39709
I20260810 03:50:22.340994 30677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:39709 every 8 connection(s)
I20260810 03:50:22.341848 30435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:50:22.343727 30391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:22.330406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:58149 (local address 127.27.153.126:46701)
0810 03:50:22.330529 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:22.330533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:22.331729 (+ 1196us) server_negotiation.cc:407] Connection header received
0810 03:50:22.332852 (+ 1123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:22.332856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:22.332922 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:22.333008 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:22.334373 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.334955 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.336221 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.336509 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.336607 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:22.339831 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:22.339834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:22.339836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:22.339867 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:22.341437 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:22.341990 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:22.341993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:22.341995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:22.342082 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:22.342466 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:22.342470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:22.342472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:22.342740 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:22.342838 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:22.343360 (+ 522us) server_negotiation.cc:299] Negotiation successful
0810 03:50:22.343548 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 03:50:22.344524 30476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:22.330995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:46701 (local address 127.27.153.65:58149)
0810 03:50:22.331563 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:22.331613 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:22.332569 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:22.333250 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:22.333257 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:22.333615 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:22.334182 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:22.334199 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.335095 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.335098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:22.336083 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:22.336096 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.336706 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.336711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:22.336873 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:22.337513 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:22.337534 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:22.339609 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:22.341567 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:22.341574 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:22.341591 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:22.341871 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:22.342168 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:22.342171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:22.342174 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:22.342346 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:22.342845 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:22.342851 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:22.343102 (+ 251us) client_negotiation.cc:769] Sending connection context
0810 03:50:22.343279 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 03:50:22.343654 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":129,"threads_started":1}
I20260810 03:50:22.347539 30679 heartbeater.cc:347] Connected to a master server at 127.27.153.126:46701
I20260810 03:50:22.347749 30679 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:22.348269 30679 heartbeater.cc:511] Master 127.27.153.126:46701 requested a full tablet report, sending...
I20260810 03:50:22.349416 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 30435
I20260810 03:50:22.349538 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:50:22.349776 30314 ts_manager.cc:194] Registered new tserver with Master: 1c2d8d5df0a74479859b89f4fb899221 (127.27.153.65:39709)
I20260810 03:50:22.349845 28261 external_mini_cluster.cc:1428] Setting key 8ff34b1e5219b7f36ec94c949775ab10
I20260810 03:50:22.351235 30314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:58149
I20260810 03:50:22.357548 30686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:22.348394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:46701 (local address 127.27.153.65:47897)
0810 03:50:22.348696 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:22.348710 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:22.348810 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:22.349052 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:22.349055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:22.349290 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:22.349419 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:22.349428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.350478 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.350481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:22.351093 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:22.351102 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.351658 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.351662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:22.351762 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:22.352310 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:22.352322 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:22.352664 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:22.355823 (+ 3159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:22.355827 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:22.355829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:22.356103 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:22.356450 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:22.356453 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:22.356455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:22.356578 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:22.357006 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:22.357010 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:22.357127 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 03:50:22.357238 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 03:50:22.357376 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":116,"threads_started":1}
I20260810 03:50:22.357596 30391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:22.348486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:47897 (local address 127.27.153.126:46701)
0810 03:50:22.348636 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:22.348640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:22.348784 (+ 144us) server_negotiation.cc:407] Connection header received
0810 03:50:22.348896 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:22.348898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:22.348940 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:22.349020 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:22.349568 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.350316 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.351210 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.351529 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.353945 (+ 2416us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:22.354086 (+ 141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:22.354090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:22.354093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:22.354109 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:22.355694 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:22.356240 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:22.356243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:22.356244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:22.356336 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:22.356698 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:22.356701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:22.356703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:22.356910 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:22.356993 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:22.357350 (+ 357us) server_negotiation.cc:299] Negotiation successful
0810 03:50:22.357474 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.27.153.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
I20260810 03:50:22.382193 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:46701
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:22.499236 30694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:22.499562 30694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:22.499639 30694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:22.503795 30694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:22.503875 30694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:22.503930 30694 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:50:22.503988 30694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:50:22.509867 30694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.153.126:46701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:22.511089 30694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:22.512238 30694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:50:22.522836 30694 server_base.cc:1034] running on GCE node
W20260810 03:50:22.522785 30703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:22.522785 30708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:22.522747 30705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:22.523416 30694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:22.524145 30694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:22.525352 30694 hybrid_clock.cc:648] HybridClock initialized: now 1786333822525334 us; error 30 us; skew 500 ppm
Aug 10 03:50:22 dist-test-slave-pww0 krb5kdc[30229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333822, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:22.528329 30694 init.cc:377] Logged in from keytab as kudu/127.27.153.66@KRBTEST.COM (short username kudu)
I20260810 03:50:22.529274 30694 webserver.cc:460] Webserver started at http://127.27.153.66:43921/ using document root <none> and password file <none>
I20260810 03:50:22.529870 30694 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:22.529945 30694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:22.530161 30694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:22.532481 30694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "97c601af14c240c498822e05c38879a4"
format_stamp: "Formatted at 2026-08-10 03:50:22 on dist-test-slave-pww0"
server_key: "fae82fa5b10d7c9627216013ebe26246"
server_key_iv: "ef54b4a8039b38a51cd39789fd73f862"
server_key_version: "encryptionkey@0"
I20260810 03:50:22.533083 30694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "97c601af14c240c498822e05c38879a4"
format_stamp: "Formatted at 2026-08-10 03:50:22 on dist-test-slave-pww0"
server_key: "fae82fa5b10d7c9627216013ebe26246"
server_key_iv: "ef54b4a8039b38a51cd39789fd73f862"
server_key_version: "encryptionkey@0"
I20260810 03:50:22.537102 30694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:50:22.539736 30716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:22.541121 30694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:22.541256 30694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "97c601af14c240c498822e05c38879a4"
format_stamp: "Formatted at 2026-08-10 03:50:22 on dist-test-slave-pww0"
server_key: "fae82fa5b10d7c9627216013ebe26246"
server_key_iv: "ef54b4a8039b38a51cd39789fd73f862"
server_key_version: "encryptionkey@0"
I20260810 03:50:22.541374 30694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:22.571080 30694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:22.574736 30694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:22.574952 30694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:22.576611 30694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:22.576668 30694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:22.576726 30694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:22.576753 30694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:50:22 dist-test-slave-pww0 krb5kdc[30229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333822, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:22.587568 30694 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:34781
I20260810 03:50:22.587635 30931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:34781 every 8 connection(s)
I20260810 03:50:22.588630 30694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:50:22.590392 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 30694
I20260810 03:50:22.590510 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:50:22.590782 28261 external_mini_cluster.cc:1428] Setting key d0c2058f9b2756bc0d0b4a39c1c8486c
I20260810 03:50:22.592145 30391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:22.577500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:60889 (local address 127.27.153.126:46701)
0810 03:50:22.577626 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:22.577629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:22.578869 (+ 1240us) server_negotiation.cc:407] Connection header received
0810 03:50:22.580223 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:22.580227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:22.580329 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:22.580424 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:22.582062 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.582824 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.584710 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.584952 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.585081 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:22.588254 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:22.588260 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:22.588262 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:22.588306 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:22.589862 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:22.590399 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:22.590403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:22.590405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:22.590510 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:22.590878 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:22.590881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:22.590883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:22.591142 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:22.591234 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:22.591868 (+ 634us) server_negotiation.cc:299] Negotiation successful
0810 03:50:22.591991 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 03:50:22.593104 30733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:22.578114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:46701 (local address 127.27.153.66:60889)
0810 03:50:22.578673 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:22.578725 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:22.579893 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:22.580677 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:22.580687 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:22.581120 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:22.581889 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:22.581914 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.582998 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.583002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:22.584559 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:22.584571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.585134 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.585139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:22.585361 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:22.586160 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:22.586184 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:22.588045 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:22.589973 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:22.589977 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:22.589988 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:22.590244 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:22.590609 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:22.590612 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:22.590614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:22.590763 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:22.591249 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:22.591269 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:22.591605 (+ 336us) client_negotiation.cc:769] Sending connection context
0810 03:50:22.591810 (+ 205us) client_negotiation.cc:240] Negotiation successful
0810 03:50:22.592164 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":158,"threads_started":1}
I20260810 03:50:22.594533 30932 heartbeater.cc:347] Connected to a master server at 127.27.153.126:46701
I20260810 03:50:22.594713 30932 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:22.595348 30932 heartbeater.cc:511] Master 127.27.153.126:46701 requested a full tablet report, sending...
I20260810 03:50:22.596681 30300 ts_manager.cc:194] Registered new tserver with Master: 97c601af14c240c498822e05c38879a4 (127.27.153.66:34781)
I20260810 03:50:22.597416 30300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:60889
WARNING: no policy specified for kudu/127.27.153.67@KRBTEST.COM; defaulting to no policy
I20260810 03:50:22.604719 30391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:22.596766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:57043 (local address 127.27.153.126:46701)
0810 03:50:22.596883 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:22.596888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:22.597208 (+ 320us) server_negotiation.cc:407] Connection header received
0810 03:50:22.597316 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:22.597320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:22.597370 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:22.597457 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:22.598039 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.598544 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.599806 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.600025 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.600102 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:22.601482 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:22.601485 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:22.601487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:22.601513 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:22.602885 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:22.603376 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:22.603379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:22.603380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:22.603485 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:22.603729 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:22.603731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:22.603732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:22.603969 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:22.604050 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:22.604477 (+ 427us) server_negotiation.cc:299] Negotiation successful
0810 03:50:22.604590 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 03:50:22.604771 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:22.596682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:46701 (local address 127.27.153.66:57043)
0810 03:50:22.597111 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:22.597127 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:22.597232 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:22.597494 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:22.597497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:22.597747 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:22.597909 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:22.597918 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.598676 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.598679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:22.599663 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:22.599676 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.600138 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.600142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:22.600241 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:22.600819 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:22.600836 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:22.601313 (+ 477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:22.603004 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:22.603008 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:22.603010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:22.603277 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:22.603563 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:22.603567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:22.603568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:22.603664 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:22.604136 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:22.604140 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:22.604251 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 03:50:22.604425 (+ 174us) client_negotiation.cc:240] Negotiation successful
0810 03:50:22.604597 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":20,"thread_start_us":137,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
I20260810 03:50:22.622586 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:46701
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:22.740698 30956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:22.740984 30956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:22.741030 30956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:22.745296 30956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:22.745390 30956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:22.745442 30956 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:50:22.745486 30956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:50:22.751387 30956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.153.126:46701
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:22.752607 30956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:22.753739 30956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:22.764761 30972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:22.764731 30968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:22.764735 30969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:22.764967 30956 server_base.cc:1034] running on GCE node
I20260810 03:50:22.765547 30956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:22.766222 30956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:22.767374 30956 hybrid_clock.cc:648] HybridClock initialized: now 1786333822767344 us; error 44 us; skew 500 ppm
Aug 10 03:50:22 dist-test-slave-pww0 krb5kdc[30229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333822, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:22.770370 30956 init.cc:377] Logged in from keytab as kudu/127.27.153.67@KRBTEST.COM (short username kudu)
I20260810 03:50:22.771360 30956 webserver.cc:460] Webserver started at http://127.27.153.67:42079/ using document root <none> and password file <none>
I20260810 03:50:22.772014 30956 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:22.772084 30956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:22.772295 30956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:22.774478 30956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "e33d0fbc1b9c456cb8ce67690ce763b0"
format_stamp: "Formatted at 2026-08-10 03:50:22 on dist-test-slave-pww0"
server_key: "4fb128ae4a9b8fd531fbd8325d728d78"
server_key_iv: "994a3e9440674f48aa4aa10e5ae1f157"
server_key_version: "encryptionkey@0"
I20260810 03:50:22.775058 30956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "e33d0fbc1b9c456cb8ce67690ce763b0"
format_stamp: "Formatted at 2026-08-10 03:50:22 on dist-test-slave-pww0"
server_key: "4fb128ae4a9b8fd531fbd8325d728d78"
server_key_iv: "994a3e9440674f48aa4aa10e5ae1f157"
server_key_version: "encryptionkey@0"
I20260810 03:50:22.779163 30956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 03:50:22.781939 30987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:22.783236 30956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 03:50:22.783351 30956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "e33d0fbc1b9c456cb8ce67690ce763b0"
format_stamp: "Formatted at 2026-08-10 03:50:22 on dist-test-slave-pww0"
server_key: "4fb128ae4a9b8fd531fbd8325d728d78"
server_key_iv: "994a3e9440674f48aa4aa10e5ae1f157"
server_key_version: "encryptionkey@0"
I20260810 03:50:22.783501 30956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:22.801957 30956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:22.805251 30956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:22.805452 30956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:22.807049 30956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:22.807109 30956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:22.807166 30956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:22.807196 30956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:50:22 dist-test-slave-pww0 krb5kdc[30229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333822, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:22.818715 30956 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:46553
I20260810 03:50:22.818760 31200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:46553 every 8 connection(s)
I20260810 03:50:22.819973 30956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:50:22.820369 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 30956
I20260810 03:50:22.820492 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:50:22.820811 28261 external_mini_cluster.cc:1428] Setting key 659b028460b1a5ff1bd1f2187758a752
I20260810 03:50:22.821787 30391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:22.807859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:44821 (local address 127.27.153.126:46701)
0810 03:50:22.807976 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:22.807980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:22.809117 (+ 1137us) server_negotiation.cc:407] Connection header received
0810 03:50:22.810236 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:22.810241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:22.810326 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:22.810376 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:22.812017 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.812878 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.814208 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.814557 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.814679 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:22.817691 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:22.817696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:22.817698 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:22.817733 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:22.819339 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:22.819921 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:22.819925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:22.819928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:22.820032 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:22.820482 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:22.820485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:22.820487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:22.820736 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:22.820832 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:22.821494 (+ 662us) server_negotiation.cc:299] Negotiation successful
0810 03:50:22.821632 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 03:50:22.822665 31003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:22.808437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:46701 (local address 127.27.153.67:44821)
0810 03:50:22.808960 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:22.809005 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:22.809880 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:22.810726 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:22.810733 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:22.811079 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:22.811841 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:22.811861 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.813005 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.813009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:22.814060 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:22.814071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.814719 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.814722 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:22.814919 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:22.815563 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:22.815585 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:22.817498 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:22.819512 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:22.819519 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:22.819536 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:22.819811 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:22.820141 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:22.820144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:22.820146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:22.820356 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:22.820892 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:22.820899 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:22.821210 (+ 311us) client_negotiation.cc:769] Sending connection context
0810 03:50:22.821412 (+ 202us) client_negotiation.cc:240] Negotiation successful
0810 03:50:22.821745 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":147,"threads_started":1}
I20260810 03:50:22.823868 31201 heartbeater.cc:347] Connected to a master server at 127.27.153.126:46701
I20260810 03:50:22.824020 31201 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:22.824549 31201 heartbeater.cc:511] Master 127.27.153.126:46701 requested a full tablet report, sending...
I20260810 03:50:22.825572 30314 ts_manager.cc:194] Registered new tserver with Master: e33d0fbc1b9c456cb8ce67690ce763b0 (127.27.153.67:46553)
I20260810 03:50:22.826275 30314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:44821
I20260810 03:50:22.832515 31210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:22.825950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:46701 (local address 127.27.153.67:48595)
0810 03:50:22.826243 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:22.826257 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:22.826363 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:22.826608 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:22.826610 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:22.826785 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:22.826904 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:22.826911 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.827686 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.827689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:22.828460 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:22.828469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.828922 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.828925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:22.828991 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:22.829423 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:22.829434 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:22.829751 (+ 317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:22.831184 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:22.831186 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:22.831188 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:22.831388 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:22.831686 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:22.831688 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:22.831689 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:22.831768 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:22.832095 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:22.832097 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:22.832173 (+ 76us) client_negotiation.cc:769] Sending connection context
0810 03:50:22.832245 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 03:50:22.832365 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":115,"threads_started":1}
I20260810 03:50:22.832515 30391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:22.826030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:48595 (local address 127.27.153.126:46701)
0810 03:50:22.826148 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:22.826152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:22.826323 (+ 171us) server_negotiation.cc:407] Connection header received
0810 03:50:22.826447 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:22.826451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:22.826520 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:22.826591 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:22.827013 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.827571 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.828586 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.828828 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.828914 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:22.829876 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:22.829879 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:22.829881 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:22.829904 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:22.831083 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:22.831521 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:22.831524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:22.831526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:22.831611 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:22.831841 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:22.831843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:22.831844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:22.832028 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:22.832105 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:22.832296 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 03:50:22.832394 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 03:50:22.834600 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 03:50:22 dist-test-slave-pww0 krb5kdc[30229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333822, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 03:50:22 dist-test-slave-pww0 krb5kdc[30229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333822, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:22.861786 30391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:22.853523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56046 (local address 127.27.153.126:46701)
0810 03:50:22.853674 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:22.853679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:22.853695 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:50:22.853741 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:22.853744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:22.853786 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:22.853865 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:22.854559 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.855089 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.855973 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.856199 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.856278 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:22.858695 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:22.858699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:22.858702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:22.858733 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:22.860067 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:22.860474 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:22.860478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:22.860479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:22.860567 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:22.860792 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:22.860795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:22.860796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:22.861034 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:22.861156 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:22.861458 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 03:50:22.861592 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 03:50:22.864951 30314 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:50:22.868050 30314 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:50:22.883284 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:22.877006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:39709 (local address 127.0.0.1:39698)
0810 03:50:22.877588 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:22.877643 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:22.877781 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:22.879547 (+ 1766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:22.879550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:22.879569 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:22.879667 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:22.879673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.880926 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.880929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:22.882005 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:22.882018 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.882834 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.882838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:22.882927 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:22.882951 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:50:22.883025 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 03:50:22.883100 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"spinlock_wait_cycles":8320,"thread_start_us":105,"threads_started":1}
I20260810 03:50:22.883302 31237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:22.877711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39698 (local address 127.27.153.65:39709)
0810 03:50:22.879063 (+ 1352us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:22.879069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:22.879092 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:50:22.879180 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:22.879185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:22.879366 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:22.879537 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:22.879782 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.880792 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.882127 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.882710 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.882818 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:22.882964 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:22.883064 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 03:50:22.883163 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1246,"thread_start_us":82,"threads_started":1}
I20260810 03:50:22.883409 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:22.877715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32808 (local address 127.27.153.67:46553)
0810 03:50:22.878269 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:22.878282 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:22.878307 (+ 25us) server_negotiation.cc:407] Connection header received
0810 03:50:22.878386 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:22.878391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:22.878606 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:22.878763 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:22.879069 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.880029 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.881819 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.882506 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.882593 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:22.882752 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:22.883188 (+ 436us) server_negotiation.cc:299] Negotiation successful
0810 03:50:22.883279 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":111,"threads_started":1}
I20260810 03:50:22.883733 31233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:22.877234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:46553 (local address 127.0.0.1:32808)
0810 03:50:22.878142 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:22.878157 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:22.878248 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:22.878777 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:22.878782 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:22.878804 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:22.878929 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:22.878936 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.880125 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.880128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:22.881698 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:22.881710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.882625 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.882629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:22.882736 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:22.882818 (+ 82us) client_negotiation.cc:769] Sending connection context
0810 03:50:22.882927 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 03:50:22.883068 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":58,"thread_start_us":102,"threads_started":1}
I20260810 03:50:22.884361 31236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:22.876905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:34781 (local address 127.0.0.1:38310)
0810 03:50:22.877588 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:22.877643 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:22.877781 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:22.878926 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:22.878929 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:22.878944 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:22.879025 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:22.879032 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.880399 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.880402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:22.881698 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:22.881710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.883981 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.883985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:22.884095 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:22.884117 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 03:50:22.884187 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 03:50:22.884260 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":66,"spinlock_wait_cycles":2560,"thread_start_us":81,"threads_started":1}
I20260810 03:50:22.884836 31235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:22.877052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38310 (local address 127.27.153.66:34781)
0810 03:50:22.878504 (+ 1452us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:22.878514 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:22.878536 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:50:22.878604 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:22.878608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:22.878796 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:22.878941 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:22.879122 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.880252 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.882819 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:22.883647 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:22.883759 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:22.883919 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:22.884591 (+ 672us) server_negotiation.cc:299] Negotiation successful
0810 03:50:22.884706 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1341,"thread_start_us":119,"threads_started":1}
I20260810 03:50:22.886868 30551 tablet_service.cc:1467] Processing CreateTablet for tablet 3ac33c35ccc44bbd95665ec797d81ce3 (DEFAULT_TABLE table=test-table [id=7ef0539f1a9a4fd0ba7eefba36b2de73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:22.887658 30818 tablet_service.cc:1467] Processing CreateTablet for tablet 3ac33c35ccc44bbd95665ec797d81ce3 (DEFAULT_TABLE table=test-table [id=7ef0539f1a9a4fd0ba7eefba36b2de73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:22.888166 30551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ac33c35ccc44bbd95665ec797d81ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:22.888785 30818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ac33c35ccc44bbd95665ec797d81ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:22.894380 31076 tablet_service.cc:1467] Processing CreateTablet for tablet 3ac33c35ccc44bbd95665ec797d81ce3 (DEFAULT_TABLE table=test-table [id=7ef0539f1a9a4fd0ba7eefba36b2de73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:22.895443 31076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ac33c35ccc44bbd95665ec797d81ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:22.896440 31239 tablet_bootstrap.cc:492] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221: Bootstrap starting.
I20260810 03:50:22.897891 31241 tablet_bootstrap.cc:492] T 3ac33c35ccc44bbd95665ec797d81ce3 P 97c601af14c240c498822e05c38879a4: Bootstrap starting.
I20260810 03:50:22.898876 31239 tablet_bootstrap.cc:654] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:22.899719 31241 tablet_bootstrap.cc:654] T 3ac33c35ccc44bbd95665ec797d81ce3 P 97c601af14c240c498822e05c38879a4: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:22.899895 31239 log.cc:824] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:22.900547 31241 log.cc:824] T 3ac33c35ccc44bbd95665ec797d81ce3 P 97c601af14c240c498822e05c38879a4: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:22.902168 31239 tablet_bootstrap.cc:492] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221: No bootstrap required, opened a new log
I20260810 03:50:22.902235 31241 tablet_bootstrap.cc:492] T 3ac33c35ccc44bbd95665ec797d81ce3 P 97c601af14c240c498822e05c38879a4: No bootstrap required, opened a new log
I20260810 03:50:22.902397 31239 ts_tablet_manager.cc:1397] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:50:22.902421 31241 ts_tablet_manager.cc:1397] T 3ac33c35ccc44bbd95665ec797d81ce3 P 97c601af14c240c498822e05c38879a4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260810 03:50:22.904295 31244 tablet_bootstrap.cc:492] T 3ac33c35ccc44bbd95665ec797d81ce3 P e33d0fbc1b9c456cb8ce67690ce763b0: Bootstrap starting.
I20260810 03:50:22.905980 31241 raft_consensus.cc:348] T 3ac33c35ccc44bbd95665ec797d81ce3 P 97c601af14c240c498822e05c38879a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:22.906284 31241 raft_consensus.cc:374] T 3ac33c35ccc44bbd95665ec797d81ce3 P 97c601af14c240c498822e05c38879a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:22.906356 31241 raft_consensus.cc:729] T 3ac33c35ccc44bbd95665ec797d81ce3 P 97c601af14c240c498822e05c38879a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c601af14c240c498822e05c38879a4, State: Initialized, Role: FOLLOWER
I20260810 03:50:22.906611 31239 raft_consensus.cc:348] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:22.906893 31239 raft_consensus.cc:374] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:22.906951 31239 raft_consensus.cc:729] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2d8d5df0a74479859b89f4fb899221, State: Initialized, Role: FOLLOWER
I20260810 03:50:22.906865 31241 consensus_queue.cc:260] T 3ac33c35ccc44bbd95665ec797d81ce3 P 97c601af14c240c498822e05c38879a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:22.907410 31239 consensus_queue.cc:260] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:22.907655 31244 tablet_bootstrap.cc:654] T 3ac33c35ccc44bbd95665ec797d81ce3 P e33d0fbc1b9c456cb8ce67690ce763b0: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:22.908001 31241 ts_tablet_manager.cc:1428] T 3ac33c35ccc44bbd95665ec797d81ce3 P 97c601af14c240c498822e05c38879a4: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260810 03:50:22.908290 31239 ts_tablet_manager.cc:1428] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 03:50:22.908402 30932 heartbeater.cc:503] Master 127.27.153.126:46701 was elected leader, sending a full tablet report...
I20260810 03:50:22.908586 31244 log.cc:824] T 3ac33c35ccc44bbd95665ec797d81ce3 P e33d0fbc1b9c456cb8ce67690ce763b0: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:22.908638 30679 heartbeater.cc:503] Master 127.27.153.126:46701 was elected leader, sending a full tablet report...
I20260810 03:50:22.910969 31244 tablet_bootstrap.cc:492] T 3ac33c35ccc44bbd95665ec797d81ce3 P e33d0fbc1b9c456cb8ce67690ce763b0: No bootstrap required, opened a new log
I20260810 03:50:22.911188 31244 ts_tablet_manager.cc:1397] T 3ac33c35ccc44bbd95665ec797d81ce3 P e33d0fbc1b9c456cb8ce67690ce763b0: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260810 03:50:22.914708 31244 raft_consensus.cc:348] T 3ac33c35ccc44bbd95665ec797d81ce3 P e33d0fbc1b9c456cb8ce67690ce763b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:22.914974 31244 raft_consensus.cc:374] T 3ac33c35ccc44bbd95665ec797d81ce3 P e33d0fbc1b9c456cb8ce67690ce763b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:22.915009 31244 raft_consensus.cc:729] T 3ac33c35ccc44bbd95665ec797d81ce3 P e33d0fbc1b9c456cb8ce67690ce763b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e33d0fbc1b9c456cb8ce67690ce763b0, State: Initialized, Role: FOLLOWER
I20260810 03:50:22.915652 31244 consensus_queue.cc:260] T 3ac33c35ccc44bbd95665ec797d81ce3 P e33d0fbc1b9c456cb8ce67690ce763b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:22.916541 31201 heartbeater.cc:503] Master 127.27.153.126:46701 was elected leader, sending a full tablet report...
I20260810 03:50:22.916873 31244 ts_tablet_manager.cc:1428] T 3ac33c35ccc44bbd95665ec797d81ce3 P e33d0fbc1b9c456cb8ce67690ce763b0: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260810 03:50:23.025020 31247 raft_consensus.cc:482] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:50:23.025291 31247 raft_consensus.cc:504] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:23.026659 31247 leader_election.cc:288] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97c601af14c240c498822e05c38879a4 (127.27.153.66:34781), e33d0fbc1b9c456cb8ce67690ce763b0 (127.27.153.67:46553)
I20260810 03:50:23.031078 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:23.027090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:36119 (local address 127.27.153.67:46553)
0810 03:50:23.027232 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:23.027237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:23.027531 (+ 294us) server_negotiation.cc:407] Connection header received
0810 03:50:23.027729 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:23.027734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:23.027779 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:23.027853 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:23.028144 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.028719 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.030113 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.030536 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.030648 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:23.030704 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:23.030878 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 03:50:23.030952 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 03:50:23.031119 31255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:23.026945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:46553 (local address 127.27.153.65:36119)
0810 03:50:23.027422 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:23.027436 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:23.027609 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:23.027878 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:23.027880 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:23.027897 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:23.028035 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:23.028041 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.028845 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.028849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:23.029985 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:23.029995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.030654 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.030657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:23.030757 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:23.030781 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:50:23.030847 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 03:50:23.030925 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":265,"spinlock_wait_cycles":41088,"thread_start_us":95,"threads_started":1}
I20260810 03:50:23.031157 31235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:23.027067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:54279 (local address 127.27.153.66:34781)
0810 03:50:23.027218 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:23.027224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:23.027556 (+ 332us) server_negotiation.cc:407] Connection header received
0810 03:50:23.027728 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:23.027730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:23.027793 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:23.027864 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:23.028181 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.028798 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.030138 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.030581 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.030705 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:23.030766 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:23.030937 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 03:50:23.031018 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 03:50:23.031179 31254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:23.026945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:34781 (local address 127.27.153.65:54279)
0810 03:50:23.027406 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:23.027419 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:23.027610 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:23.027891 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:23.027894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:23.027912 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:23.028053 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:23.028062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.028899 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.028903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:23.030002 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:23.030013 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.030706 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.030710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:23.030818 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:23.030838 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 03:50:23.030912 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 03:50:23.030989 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":29,"spinlock_wait_cycles":6144,"thread_start_us":154,"threads_started":1}
I20260810 03:50:23.031839 31112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ac33c35ccc44bbd95665ec797d81ce3" candidate_uuid: "1c2d8d5df0a74479859b89f4fb899221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" is_pre_election: true
I20260810 03:50:23.031877 30853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ac33c35ccc44bbd95665ec797d81ce3" candidate_uuid: "1c2d8d5df0a74479859b89f4fb899221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c601af14c240c498822e05c38879a4" is_pre_election: true
I20260810 03:50:23.032152 31112 raft_consensus.cc:2454] T 3ac33c35ccc44bbd95665ec797d81ce3 P e33d0fbc1b9c456cb8ce67690ce763b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2d8d5df0a74479859b89f4fb899221 in term 0.
I20260810 03:50:23.032173 30853 raft_consensus.cc:2454] T 3ac33c35ccc44bbd95665ec797d81ce3 P 97c601af14c240c498822e05c38879a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2d8d5df0a74479859b89f4fb899221 in term 0.
I20260810 03:50:23.032626 30466 leader_election.cc:302] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2d8d5df0a74479859b89f4fb899221, e33d0fbc1b9c456cb8ce67690ce763b0; no voters:
I20260810 03:50:23.032932 31247 raft_consensus.cc:2781] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:50:23.033027 31247 raft_consensus.cc:482] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:50:23.033078 31247 raft_consensus.cc:3037] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:23.034166 31247 raft_consensus.cc:504] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:23.034520 31247 leader_election.cc:288] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221 [CANDIDATE]: Term 1 election: Requested vote from peers 97c601af14c240c498822e05c38879a4 (127.27.153.66:34781), e33d0fbc1b9c456cb8ce67690ce763b0 (127.27.153.67:46553)
I20260810 03:50:23.034909 30853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ac33c35ccc44bbd95665ec797d81ce3" candidate_uuid: "1c2d8d5df0a74479859b89f4fb899221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c601af14c240c498822e05c38879a4"
I20260810 03:50:23.034965 31112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ac33c35ccc44bbd95665ec797d81ce3" candidate_uuid: "1c2d8d5df0a74479859b89f4fb899221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0"
I20260810 03:50:23.035029 30853 raft_consensus.cc:3037] T 3ac33c35ccc44bbd95665ec797d81ce3 P 97c601af14c240c498822e05c38879a4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:23.035106 31112 raft_consensus.cc:3037] T 3ac33c35ccc44bbd95665ec797d81ce3 P e33d0fbc1b9c456cb8ce67690ce763b0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:23.036604 30853 raft_consensus.cc:2454] T 3ac33c35ccc44bbd95665ec797d81ce3 P 97c601af14c240c498822e05c38879a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2d8d5df0a74479859b89f4fb899221 in term 1.
I20260810 03:50:23.036614 31112 raft_consensus.cc:2454] T 3ac33c35ccc44bbd95665ec797d81ce3 P e33d0fbc1b9c456cb8ce67690ce763b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2d8d5df0a74479859b89f4fb899221 in term 1.
I20260810 03:50:23.036950 30462 leader_election.cc:302] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2d8d5df0a74479859b89f4fb899221, 97c601af14c240c498822e05c38879a4; no voters:
I20260810 03:50:23.037127 31247 raft_consensus.cc:2781] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:23.037350 31247 raft_consensus.cc:686] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2d8d5df0a74479859b89f4fb899221, State: Running, Role: LEADER
I20260810 03:50:23.037688 31247 consensus_queue.cc:237] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:23.041167 30300 catalog_manager.cc:5495] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2d8d5df0a74479859b89f4fb899221 (127.27.153.65). New cstate: current_term: 1 leader_uuid: "1c2d8d5df0a74479859b89f4fb899221" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } health_report { overall_health: UNKNOWN } } }
W20260810 03:50:23.071197 31202 tablet.cc:2369] T 3ac33c35ccc44bbd95665ec797d81ce3 P e33d0fbc1b9c456cb8ce67690ce763b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:50:23.083906 31237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:23.079675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39704 (local address 127.27.153.65:39709)
0810 03:50:23.079818 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:23.079822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:23.079841 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:50:23.080150 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:23.080154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:23.080227 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:23.080330 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:50:23.080596 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.081161 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.082179 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.082436 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.082531 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:23.082921 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:50:23.083493 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:50:23.083604 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:23.083715 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 03:50:23.083781 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 03:50:23.089977 30933 tablet.cc:2369] T 3ac33c35ccc44bbd95665ec797d81ce3 P 97c601af14c240c498822e05c38879a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:50:23.090929 31112 raft_consensus.cc:1261] T 3ac33c35ccc44bbd95665ec797d81ce3 P e33d0fbc1b9c456cb8ce67690ce763b0 [term 1 FOLLOWER]: Refusing update from remote peer 1c2d8d5df0a74479859b89f4fb899221: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:23.091768 31247 consensus_queue.cc:1035] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:23.092598 30853 raft_consensus.cc:1261] T 3ac33c35ccc44bbd95665ec797d81ce3 P 97c601af14c240c498822e05c38879a4 [term 1 FOLLOWER]: Refusing update from remote peer 1c2d8d5df0a74479859b89f4fb899221: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:23.093184 31247 consensus_queue.cc:1035] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 03:50:23.093262 30681 tablet.cc:2369] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:50:23.114760 30314 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:23.115115 30314 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:23.118386 30314 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0: Sending DeleteTablet for 3 replicas of tablet 3ac33c35ccc44bbd95665ec797d81ce3
I20260810 03:50:23.119432 30551 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ac33c35ccc44bbd95665ec797d81ce3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:23 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:39698
I20260810 03:50:23.119438 31076 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ac33c35ccc44bbd95665ec797d81ce3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:23 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:32808
I20260810 03:50:23.119423 30818 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ac33c35ccc44bbd95665ec797d81ce3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:23 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:38310
I20260810 03:50:23.120522 30314 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:50:23.121032 30314 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:50:23.122301 31293 tablet_replica.cc:330] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221: stopping tablet replica
I20260810 03:50:23.122418 31291 tablet_replica.cc:330] T 3ac33c35ccc44bbd95665ec797d81ce3 P 97c601af14c240c498822e05c38879a4: stopping tablet replica
I20260810 03:50:23.122605 31296 tablet_replica.cc:330] T 3ac33c35ccc44bbd95665ec797d81ce3 P e33d0fbc1b9c456cb8ce67690ce763b0: stopping tablet replica
I20260810 03:50:23.122882 31293 raft_consensus.cc:2229] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:50:23.122882 31291 raft_consensus.cc:2229] T 3ac33c35ccc44bbd95665ec797d81ce3 P 97c601af14c240c498822e05c38879a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:23.123009 31296 raft_consensus.cc:2229] T 3ac33c35ccc44bbd95665ec797d81ce3 P e33d0fbc1b9c456cb8ce67690ce763b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:23.123325 31291 raft_consensus.cc:2258] T 3ac33c35ccc44bbd95665ec797d81ce3 P 97c601af14c240c498822e05c38879a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:23.123382 31296 raft_consensus.cc:2258] T 3ac33c35ccc44bbd95665ec797d81ce3 P e33d0fbc1b9c456cb8ce67690ce763b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:23.123641 31293 raft_consensus.cc:2258] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:23.125633 31291 ts_tablet_manager.cc:1905] T 3ac33c35ccc44bbd95665ec797d81ce3 P 97c601af14c240c498822e05c38879a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:23.127517 31296 ts_tablet_manager.cc:1905] T 3ac33c35ccc44bbd95665ec797d81ce3 P e33d0fbc1b9c456cb8ce67690ce763b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:23.128733 31291 ts_tablet_manager.cc:1918] T 3ac33c35ccc44bbd95665ec797d81ce3 P 97c601af14c240c498822e05c38879a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:50:23.128882 31291 log.cc:1196] T 3ac33c35ccc44bbd95665ec797d81ce3 P 97c601af14c240c498822e05c38879a4: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/3ac33c35ccc44bbd95665ec797d81ce3
I20260810 03:50:23.129266 31291 ts_tablet_manager.cc:1939] T 3ac33c35ccc44bbd95665ec797d81ce3 P 97c601af14c240c498822e05c38879a4: Deleting consensus metadata
I20260810 03:50:23.130888 31296 ts_tablet_manager.cc:1918] T 3ac33c35ccc44bbd95665ec797d81ce3 P e33d0fbc1b9c456cb8ce67690ce763b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:50:23.131120 31296 log.cc:1196] T 3ac33c35ccc44bbd95665ec797d81ce3 P e33d0fbc1b9c456cb8ce67690ce763b0: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/3ac33c35ccc44bbd95665ec797d81ce3
I20260810 03:50:23.132226 30278 catalog_manager.cc:4841] TS 97c601af14c240c498822e05c38879a4 (127.27.153.66:34781): tablet 3ac33c35ccc44bbd95665ec797d81ce3 (table test-table [id=7ef0539f1a9a4fd0ba7eefba36b2de73]) successfully deleted
I20260810 03:50:23.133184 31293 ts_tablet_manager.cc:1905] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:23.134662 31296 ts_tablet_manager.cc:1939] T 3ac33c35ccc44bbd95665ec797d81ce3 P e33d0fbc1b9c456cb8ce67690ce763b0: Deleting consensus metadata
I20260810 03:50:23.136301 30284 catalog_manager.cc:4841] TS e33d0fbc1b9c456cb8ce67690ce763b0 (127.27.153.67:46553): tablet 3ac33c35ccc44bbd95665ec797d81ce3 (table test-table [id=7ef0539f1a9a4fd0ba7eefba36b2de73]) successfully deleted
I20260810 03:50:23.136670 31076 tablet_service.cc:1467] Processing CreateTablet for tablet 38e105a87d544b7a938cfbee16c4638a (DEFAULT_TABLE table=test-table [id=0047bd2f27bf41f6825fd27d10ab997a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:23.136670 30551 tablet_service.cc:1467] Processing CreateTablet for tablet 38e105a87d544b7a938cfbee16c4638a (DEFAULT_TABLE table=test-table [id=0047bd2f27bf41f6825fd27d10ab997a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:23.136965 31076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38e105a87d544b7a938cfbee16c4638a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:23.136965 30551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38e105a87d544b7a938cfbee16c4638a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:23.136982 30818 tablet_service.cc:1467] Processing CreateTablet for tablet 38e105a87d544b7a938cfbee16c4638a (DEFAULT_TABLE table=test-table [id=0047bd2f27bf41f6825fd27d10ab997a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:23.137266 30818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38e105a87d544b7a938cfbee16c4638a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:23.139585 31239 tablet_bootstrap.cc:492] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221: Bootstrap starting.
I20260810 03:50:23.139575 31244 tablet_bootstrap.cc:492] T 38e105a87d544b7a938cfbee16c4638a P e33d0fbc1b9c456cb8ce67690ce763b0: Bootstrap starting.
I20260810 03:50:23.139707 31241 tablet_bootstrap.cc:492] T 38e105a87d544b7a938cfbee16c4638a P 97c601af14c240c498822e05c38879a4: Bootstrap starting.
I20260810 03:50:23.140687 31241 tablet_bootstrap.cc:654] T 38e105a87d544b7a938cfbee16c4638a P 97c601af14c240c498822e05c38879a4: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:23.140689 31239 tablet_bootstrap.cc:654] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:23.140689 31244 tablet_bootstrap.cc:654] T 38e105a87d544b7a938cfbee16c4638a P e33d0fbc1b9c456cb8ce67690ce763b0: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:23.142417 31293 ts_tablet_manager.cc:1918] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:50:23.142561 31293 log.cc:1196] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-0/wal/wals/3ac33c35ccc44bbd95665ec797d81ce3
I20260810 03:50:23.142704 31244 tablet_bootstrap.cc:492] T 38e105a87d544b7a938cfbee16c4638a P e33d0fbc1b9c456cb8ce67690ce763b0: No bootstrap required, opened a new log
I20260810 03:50:23.142795 31244 ts_tablet_manager.cc:1397] T 38e105a87d544b7a938cfbee16c4638a P e33d0fbc1b9c456cb8ce67690ce763b0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:50:23.142889 31241 tablet_bootstrap.cc:492] T 38e105a87d544b7a938cfbee16c4638a P 97c601af14c240c498822e05c38879a4: No bootstrap required, opened a new log
I20260810 03:50:23.142957 31293 ts_tablet_manager.cc:1939] T 3ac33c35ccc44bbd95665ec797d81ce3 P 1c2d8d5df0a74479859b89f4fb899221: Deleting consensus metadata
I20260810 03:50:23.142966 31241 ts_tablet_manager.cc:1397] T 38e105a87d544b7a938cfbee16c4638a P 97c601af14c240c498822e05c38879a4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:50:23.143368 31244 raft_consensus.cc:348] T 38e105a87d544b7a938cfbee16c4638a P e33d0fbc1b9c456cb8ce67690ce763b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:23.143548 31244 raft_consensus.cc:374] T 38e105a87d544b7a938cfbee16c4638a P e33d0fbc1b9c456cb8ce67690ce763b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:23.143420 31241 raft_consensus.cc:348] T 38e105a87d544b7a938cfbee16c4638a P 97c601af14c240c498822e05c38879a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:23.143589 31244 raft_consensus.cc:729] T 38e105a87d544b7a938cfbee16c4638a P e33d0fbc1b9c456cb8ce67690ce763b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e33d0fbc1b9c456cb8ce67690ce763b0, State: Initialized, Role: FOLLOWER
I20260810 03:50:23.143594 31241 raft_consensus.cc:374] T 38e105a87d544b7a938cfbee16c4638a P 97c601af14c240c498822e05c38879a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:23.143625 31241 raft_consensus.cc:729] T 38e105a87d544b7a938cfbee16c4638a P 97c601af14c240c498822e05c38879a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c601af14c240c498822e05c38879a4, State: Initialized, Role: FOLLOWER
I20260810 03:50:23.143731 31244 consensus_queue.cc:260] T 38e105a87d544b7a938cfbee16c4638a P e33d0fbc1b9c456cb8ce67690ce763b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:23.143749 31241 consensus_queue.cc:260] T 38e105a87d544b7a938cfbee16c4638a P 97c601af14c240c498822e05c38879a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:23.144002 31241 ts_tablet_manager.cc:1428] T 38e105a87d544b7a938cfbee16c4638a P 97c601af14c240c498822e05c38879a4: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260810 03:50:23.144025 31244 ts_tablet_manager.cc:1428] T 38e105a87d544b7a938cfbee16c4638a P e33d0fbc1b9c456cb8ce67690ce763b0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:50:23.144519 30283 catalog_manager.cc:4841] TS 1c2d8d5df0a74479859b89f4fb899221 (127.27.153.65:39709): tablet 3ac33c35ccc44bbd95665ec797d81ce3 (table test-table [id=7ef0539f1a9a4fd0ba7eefba36b2de73]) successfully deleted
I20260810 03:50:23.146131 31239 tablet_bootstrap.cc:492] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221: No bootstrap required, opened a new log
I20260810 03:50:23.146267 31239 ts_tablet_manager.cc:1397] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260810 03:50:23.147027 31239 raft_consensus.cc:348] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:23.147151 31239 raft_consensus.cc:374] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:23.147185 31239 raft_consensus.cc:729] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2d8d5df0a74479859b89f4fb899221, State: Initialized, Role: FOLLOWER
I20260810 03:50:23.147332 31239 consensus_queue.cc:260] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:23.147727 31239 ts_tablet_manager.cc:1428] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:50:23.202077 31256 raft_consensus.cc:482] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:50:23.202234 31256 raft_consensus.cc:504] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:23.202687 31256 leader_election.cc:288] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97c601af14c240c498822e05c38879a4 (127.27.153.66:34781), e33d0fbc1b9c456cb8ce67690ce763b0 (127.27.153.67:46553)
I20260810 03:50:23.203177 31112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e105a87d544b7a938cfbee16c4638a" candidate_uuid: "1c2d8d5df0a74479859b89f4fb899221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" is_pre_election: true
I20260810 03:50:23.203163 30853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e105a87d544b7a938cfbee16c4638a" candidate_uuid: "1c2d8d5df0a74479859b89f4fb899221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c601af14c240c498822e05c38879a4" is_pre_election: true
I20260810 03:50:23.203330 31112 raft_consensus.cc:2454] T 38e105a87d544b7a938cfbee16c4638a P e33d0fbc1b9c456cb8ce67690ce763b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2d8d5df0a74479859b89f4fb899221 in term 0.
I20260810 03:50:23.203330 30853 raft_consensus.cc:2454] T 38e105a87d544b7a938cfbee16c4638a P 97c601af14c240c498822e05c38879a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c2d8d5df0a74479859b89f4fb899221 in term 0.
I20260810 03:50:23.203663 30462 leader_election.cc:302] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2d8d5df0a74479859b89f4fb899221, 97c601af14c240c498822e05c38879a4; no voters:
I20260810 03:50:23.203850 31256 raft_consensus.cc:2781] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:50:23.203912 31256 raft_consensus.cc:482] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:50:23.203945 31256 raft_consensus.cc:3037] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:23.204933 31256 raft_consensus.cc:504] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:23.205302 31256 leader_election.cc:288] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221 [CANDIDATE]: Term 1 election: Requested vote from peers 97c601af14c240c498822e05c38879a4 (127.27.153.66:34781), e33d0fbc1b9c456cb8ce67690ce763b0 (127.27.153.67:46553)
I20260810 03:50:23.205714 31112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e105a87d544b7a938cfbee16c4638a" candidate_uuid: "1c2d8d5df0a74479859b89f4fb899221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0"
I20260810 03:50:23.205724 30853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38e105a87d544b7a938cfbee16c4638a" candidate_uuid: "1c2d8d5df0a74479859b89f4fb899221" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c601af14c240c498822e05c38879a4"
I20260810 03:50:23.205838 31112 raft_consensus.cc:3037] T 38e105a87d544b7a938cfbee16c4638a P e33d0fbc1b9c456cb8ce67690ce763b0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:23.205873 30853 raft_consensus.cc:3037] T 38e105a87d544b7a938cfbee16c4638a P 97c601af14c240c498822e05c38879a4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:23.206794 31112 raft_consensus.cc:2454] T 38e105a87d544b7a938cfbee16c4638a P e33d0fbc1b9c456cb8ce67690ce763b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2d8d5df0a74479859b89f4fb899221 in term 1.
I20260810 03:50:23.207106 30466 leader_election.cc:302] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c2d8d5df0a74479859b89f4fb899221, e33d0fbc1b9c456cb8ce67690ce763b0; no voters:
I20260810 03:50:23.207116 30853 raft_consensus.cc:2454] T 38e105a87d544b7a938cfbee16c4638a P 97c601af14c240c498822e05c38879a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c2d8d5df0a74479859b89f4fb899221 in term 1.
I20260810 03:50:23.207309 31256 raft_consensus.cc:2781] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:23.207396 31256 raft_consensus.cc:686] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221 [term 1 LEADER]: Becoming Leader. State: Replica: 1c2d8d5df0a74479859b89f4fb899221, State: Running, Role: LEADER
I20260810 03:50:23.207574 31256 consensus_queue.cc:237] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:23.209277 30313 catalog_manager.cc:5495] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c2d8d5df0a74479859b89f4fb899221 (127.27.153.65). New cstate: current_term: 1 leader_uuid: "1c2d8d5df0a74479859b89f4fb899221" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } health_report { overall_health: UNKNOWN } } }
Aug 10 03:50:23 dist-test-slave-pww0 krb5kdc[30229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333821, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:23.255316 31312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:23.245425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:46701 (local address 127.0.0.1:56056)
0810 03:50:23.245791 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:23.245807 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:23.245927 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:23.246274 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:23.246276 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:23.246710 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:23.246814 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:23.246821 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.247959 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.247962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:23.248541 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:23.248549 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.249140 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.249143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:23.249210 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:23.249856 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:23.249868 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:23.251698 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:23.253556 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:23.253569 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:23.253577 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:23.253904 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:23.254210 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:23.254213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:23.254215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:23.254305 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:23.254660 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:23.254667 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:23.254807 (+ 140us) client_negotiation.cc:769] Sending connection context
0810 03:50:23.254918 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 03:50:23.255099 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":149,"threads_started":1}
I20260810 03:50:23.255331 30391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:23.245483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56056 (local address 127.27.153.126:46701)
0810 03:50:23.245643 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:23.245648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:23.245896 (+ 248us) server_negotiation.cc:407] Connection header received
0810 03:50:23.246005 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:23.246009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:23.246098 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:23.246211 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:23.246982 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.247831 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.248661 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.249001 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.249137 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:23.251912 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:23.251916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:23.251918 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:23.251945 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:23.253430 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:23.254027 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:23.254030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:23.254031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:23.254125 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:23.254377 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:23.254379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:23.254380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:23.254570 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:23.254645 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:23.254963 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 03:50:23.255171 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":51}
I20260810 03:50:23.259891 30313 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:56056:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 03:50:23.260713 30313 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:50:23.265918 30551 tablet_service.cc:1467] Processing CreateTablet for tablet 2d166dd7bad349c6981a1513060d77a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7de3414a0bf4442e9771aab5300f3506]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:50:23.266026 31076 tablet_service.cc:1467] Processing CreateTablet for tablet 2d166dd7bad349c6981a1513060d77a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7de3414a0bf4442e9771aab5300f3506]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:50:23.266494 31076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d166dd7bad349c6981a1513060d77a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:23.266495 30551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d166dd7bad349c6981a1513060d77a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:23.267935 30818 tablet_service.cc:1467] Processing CreateTablet for tablet 2d166dd7bad349c6981a1513060d77a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7de3414a0bf4442e9771aab5300f3506]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:50:23.268359 30818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d166dd7bad349c6981a1513060d77a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:23.269483 31239 tablet_bootstrap.cc:492] T 2d166dd7bad349c6981a1513060d77a9 P 1c2d8d5df0a74479859b89f4fb899221: Bootstrap starting.
I20260810 03:50:23.269742 31244 tablet_bootstrap.cc:492] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0: Bootstrap starting.
I20260810 03:50:23.270401 31239 tablet_bootstrap.cc:654] T 2d166dd7bad349c6981a1513060d77a9 P 1c2d8d5df0a74479859b89f4fb899221: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:23.270771 31244 tablet_bootstrap.cc:654] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:23.271831 31239 tablet_bootstrap.cc:492] T 2d166dd7bad349c6981a1513060d77a9 P 1c2d8d5df0a74479859b89f4fb899221: No bootstrap required, opened a new log
I20260810 03:50:23.271922 31239 ts_tablet_manager.cc:1397] T 2d166dd7bad349c6981a1513060d77a9 P 1c2d8d5df0a74479859b89f4fb899221: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 03:50:23.272397 31244 tablet_bootstrap.cc:492] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0: No bootstrap required, opened a new log
I20260810 03:50:23.272375 31239 raft_consensus.cc:348] T 2d166dd7bad349c6981a1513060d77a9 P 1c2d8d5df0a74479859b89f4fb899221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:23.272480 31244 ts_tablet_manager.cc:1397] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:50:23.272495 31239 raft_consensus.cc:374] T 2d166dd7bad349c6981a1513060d77a9 P 1c2d8d5df0a74479859b89f4fb899221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:23.272526 31239 raft_consensus.cc:729] T 2d166dd7bad349c6981a1513060d77a9 P 1c2d8d5df0a74479859b89f4fb899221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c2d8d5df0a74479859b89f4fb899221, State: Initialized, Role: FOLLOWER
I20260810 03:50:23.272650 31239 consensus_queue.cc:260] T 2d166dd7bad349c6981a1513060d77a9 P 1c2d8d5df0a74479859b89f4fb899221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:23.272925 31239 ts_tablet_manager.cc:1428] T 2d166dd7bad349c6981a1513060d77a9 P 1c2d8d5df0a74479859b89f4fb899221: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 03:50:23.272882 31244 raft_consensus.cc:348] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:23.272982 31244 raft_consensus.cc:374] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:23.273008 31244 raft_consensus.cc:729] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e33d0fbc1b9c456cb8ce67690ce763b0, State: Initialized, Role: FOLLOWER
I20260810 03:50:23.273013 31241 tablet_bootstrap.cc:492] T 2d166dd7bad349c6981a1513060d77a9 P 97c601af14c240c498822e05c38879a4: Bootstrap starting.
I20260810 03:50:23.273129 31244 consensus_queue.cc:260] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:23.273370 31244 ts_tablet_manager.cc:1428] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:50:23.273416 31256 tablet_replica.cc:439] T 2d166dd7bad349c6981a1513060d77a9 P 1c2d8d5df0a74479859b89f4fb899221: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } } }
I20260810 03:50:23.273572 31256 tablet_replica.cc:442] T 2d166dd7bad349c6981a1513060d77a9 P 1c2d8d5df0a74479859b89f4fb899221: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:50:23.273590 31251 tablet_replica.cc:439] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } } }
I20260810 03:50:23.273741 31251 tablet_replica.cc:442] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:50:23.274065 31241 tablet_bootstrap.cc:654] T 2d166dd7bad349c6981a1513060d77a9 P 97c601af14c240c498822e05c38879a4: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:23.275449 31241 tablet_bootstrap.cc:492] T 2d166dd7bad349c6981a1513060d77a9 P 97c601af14c240c498822e05c38879a4: No bootstrap required, opened a new log
I20260810 03:50:23.275581 31241 ts_tablet_manager.cc:1397] T 2d166dd7bad349c6981a1513060d77a9 P 97c601af14c240c498822e05c38879a4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260810 03:50:23.276022 31241 raft_consensus.cc:348] T 2d166dd7bad349c6981a1513060d77a9 P 97c601af14c240c498822e05c38879a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:23.276116 31241 raft_consensus.cc:374] T 2d166dd7bad349c6981a1513060d77a9 P 97c601af14c240c498822e05c38879a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:23.276136 31241 raft_consensus.cc:729] T 2d166dd7bad349c6981a1513060d77a9 P 97c601af14c240c498822e05c38879a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c601af14c240c498822e05c38879a4, State: Initialized, Role: FOLLOWER
I20260810 03:50:23.276235 31241 consensus_queue.cc:260] T 2d166dd7bad349c6981a1513060d77a9 P 97c601af14c240c498822e05c38879a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:23.276430 31241 ts_tablet_manager.cc:1428] T 2d166dd7bad349c6981a1513060d77a9 P 97c601af14c240c498822e05c38879a4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 03:50:23.276464 31246 tablet_replica.cc:439] T 2d166dd7bad349c6981a1513060d77a9 P 97c601af14c240c498822e05c38879a4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } } }
I20260810 03:50:23.276626 31246 tablet_replica.cc:442] T 2d166dd7bad349c6981a1513060d77a9 P 97c601af14c240c498822e05c38879a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:50:23.406389 31251 raft_consensus.cc:482] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:50:23.406584 31251 raft_consensus.cc:504] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:23.407781 31251 leader_election.cc:288] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c2d8d5df0a74479859b89f4fb899221 (127.27.153.65:39709), 97c601af14c240c498822e05c38879a4 (127.27.153.66:34781)
I20260810 03:50:23.412879 31235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:23.408168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:42387 (local address 127.27.153.66:34781)
0810 03:50:23.408311 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:23.408316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:23.408527 (+ 211us) server_negotiation.cc:407] Connection header received
0810 03:50:23.408717 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:23.408720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:23.408777 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:23.408846 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:23.409079 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.409860 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.411594 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.412211 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.412339 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:23.412395 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:23.412661 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 03:50:23.412759 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:50:23.412879 31237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:23.408153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:43819 (local address 127.27.153.65:39709)
0810 03:50:23.408288 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:23.408294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:23.408483 (+ 189us) server_negotiation.cc:407] Connection header received
0810 03:50:23.408627 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:23.408629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:23.408679 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:23.408754 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:23.409078 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.409860 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.411605 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.412212 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.412339 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:23.412395 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:23.412661 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 03:50:23.412744 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:50:23.412917 31317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:23.408073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:34781 (local address 127.27.153.67:42387)
0810 03:50:23.408442 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:23.408453 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:23.408555 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:23.408863 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:23.408874 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:23.408887 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:23.408973 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:23.408978 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.409963 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.409966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:23.411439 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:23.411468 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.412330 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.412334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:23.412522 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:23.412560 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 03:50:23.412631 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 03:50:23.412723 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":118,"thread_start_us":73,"threads_started":1}
I20260810 03:50:23.412917 31316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:23.408060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:39709 (local address 127.27.153.67:43819)
0810 03:50:23.408393 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:23.408404 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:23.408521 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:23.408789 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:23.408791 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:23.408808 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:23.408944 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:23.408952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.409961 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.409965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:23.411439 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:23.411452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.412330 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.412334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:23.412506 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:23.412554 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 03:50:23.412631 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 03:50:23.412723 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":155,"thread_start_us":81,"threads_started":1}
I20260810 03:50:23.413468 30853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d166dd7bad349c6981a1513060d77a9" candidate_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c601af14c240c498822e05c38879a4" is_pre_election: true
I20260810 03:50:23.413640 30853 raft_consensus.cc:2454] T 2d166dd7bad349c6981a1513060d77a9 P 97c601af14c240c498822e05c38879a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e33d0fbc1b9c456cb8ce67690ce763b0 in term 0.
I20260810 03:50:23.413832 30596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d166dd7bad349c6981a1513060d77a9" candidate_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2d8d5df0a74479859b89f4fb899221" is_pre_election: true
I20260810 03:50:23.414003 30988 leader_election.cc:302] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97c601af14c240c498822e05c38879a4, e33d0fbc1b9c456cb8ce67690ce763b0; no voters:
I20260810 03:50:23.414112 30596 raft_consensus.cc:2454] T 2d166dd7bad349c6981a1513060d77a9 P 1c2d8d5df0a74479859b89f4fb899221 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e33d0fbc1b9c456cb8ce67690ce763b0 in term 0.
I20260810 03:50:23.414287 31251 raft_consensus.cc:2781] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:50:23.414364 31251 raft_consensus.cc:482] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:50:23.414413 31251 raft_consensus.cc:3037] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:23.415447 31251 raft_consensus.cc:504] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:23.415849 31251 leader_election.cc:288] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0 [CANDIDATE]: Term 1 election: Requested vote from peers 1c2d8d5df0a74479859b89f4fb899221 (127.27.153.65:39709), 97c601af14c240c498822e05c38879a4 (127.27.153.66:34781)
I20260810 03:50:23.416213 30596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d166dd7bad349c6981a1513060d77a9" candidate_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c2d8d5df0a74479859b89f4fb899221"
I20260810 03:50:23.416227 30853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d166dd7bad349c6981a1513060d77a9" candidate_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c601af14c240c498822e05c38879a4"
I20260810 03:50:23.416328 30853 raft_consensus.cc:3037] T 2d166dd7bad349c6981a1513060d77a9 P 97c601af14c240c498822e05c38879a4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:23.416329 30596 raft_consensus.cc:3037] T 2d166dd7bad349c6981a1513060d77a9 P 1c2d8d5df0a74479859b89f4fb899221 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:23.417194 30853 raft_consensus.cc:2454] T 2d166dd7bad349c6981a1513060d77a9 P 97c601af14c240c498822e05c38879a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e33d0fbc1b9c456cb8ce67690ce763b0 in term 1.
I20260810 03:50:23.417354 30596 raft_consensus.cc:2454] T 2d166dd7bad349c6981a1513060d77a9 P 1c2d8d5df0a74479859b89f4fb899221 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e33d0fbc1b9c456cb8ce67690ce763b0 in term 1.
I20260810 03:50:23.417479 30988 leader_election.cc:302] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97c601af14c240c498822e05c38879a4, e33d0fbc1b9c456cb8ce67690ce763b0; no voters:
I20260810 03:50:23.417690 31251 raft_consensus.cc:2781] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:23.417840 31251 raft_consensus.cc:686] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0 [term 1 LEADER]: Becoming Leader. State: Replica: e33d0fbc1b9c456cb8ce67690ce763b0, State: Running, Role: LEADER
I20260810 03:50:23.418074 31251 consensus_queue.cc:237] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } }
I20260810 03:50:23.419528 31320 tablet_replica.cc:439] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0: TxnStatusTablet state changed. Reason: New leader e33d0fbc1b9c456cb8ce67690ce763b0. Latest consensus state: current_term: 1 leader_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } } }
I20260810 03:50:23.419642 31320 tablet_replica.cc:442] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:50:23.420403 31322 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 03:50:23.420610 31322 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 03:50:23.420907 30313 catalog_manager.cc:5495] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to e33d0fbc1b9c456cb8ce67690ce763b0 (127.27.153.67). New cstate: current_term: 1 leader_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } health_report { overall_health: HEALTHY } } }
I20260810 03:50:23.437371 31312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:23.433318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:46553 (local address 127.0.0.1:32818)
0810 03:50:23.433586 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:23.433600 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:23.433716 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:23.434050 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:23.434053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:23.434060 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:50:23.434171 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:23.434177 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.434878 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.434881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:23.435554 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:23.435566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.436001 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.436004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:23.436100 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:23.436149 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:50:23.437041 (+ 892us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:50:23.437060 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 03:50:23.437128 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 03:50:23.437232 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260810 03:50:23.437356 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:23.433419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32818 (local address 127.27.153.67:46553)
0810 03:50:23.433541 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:23.433545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:23.433656 (+ 111us) server_negotiation.cc:407] Connection header received
0810 03:50:23.433875 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:23.433878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:23.433937 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:23.434024 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:50:23.434274 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.434770 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.435674 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.435897 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.435986 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:23.436297 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:50:23.436913 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:50:23.437049 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:23.437141 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 03:50:23.437202 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 03:50:23.440619 30853 raft_consensus.cc:1261] T 2d166dd7bad349c6981a1513060d77a9 P 97c601af14c240c498822e05c38879a4 [term 1 FOLLOWER]: Refusing update from remote peer e33d0fbc1b9c456cb8ce67690ce763b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:23.440888 30596 raft_consensus.cc:1261] T 2d166dd7bad349c6981a1513060d77a9 P 1c2d8d5df0a74479859b89f4fb899221 [term 1 FOLLOWER]: Refusing update from remote peer e33d0fbc1b9c456cb8ce67690ce763b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:23.441179 31320 consensus_queue.cc:1035] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:23.441445 31251 consensus_queue.cc:1035] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:23.443229 31256 tablet_replica.cc:439] T 2d166dd7bad349c6981a1513060d77a9 P 1c2d8d5df0a74479859b89f4fb899221: TxnStatusTablet state changed. Reason: New leader e33d0fbc1b9c456cb8ce67690ce763b0. Latest consensus state: current_term: 1 leader_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } } }
I20260810 03:50:23.443264 31246 tablet_replica.cc:439] T 2d166dd7bad349c6981a1513060d77a9 P 97c601af14c240c498822e05c38879a4: TxnStatusTablet state changed. Reason: New leader e33d0fbc1b9c456cb8ce67690ce763b0. Latest consensus state: current_term: 1 leader_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } } }
I20260810 03:50:23.443357 31256 tablet_replica.cc:442] T 2d166dd7bad349c6981a1513060d77a9 P 1c2d8d5df0a74479859b89f4fb899221: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:50:23.443384 31246 tablet_replica.cc:442] T 2d166dd7bad349c6981a1513060d77a9 P 97c601af14c240c498822e05c38879a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:50:23.446761 31273 mvcc.cc:205] Tried to move back new op lower bound from 7316823340807364608 to 7316823340723019776. Current Snapshot: MvccSnapshot[applied={T|T < 7316823340807364608}]
I20260810 03:50:23.446967 31246 tablet_replica.cc:439] T 2d166dd7bad349c6981a1513060d77a9 P 97c601af14c240c498822e05c38879a4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } } }
I20260810 03:50:23.447078 31246 tablet_replica.cc:442] T 2d166dd7bad349c6981a1513060d77a9 P 97c601af14c240c498822e05c38879a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:50:23.448122 31251 tablet_replica.cc:439] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } } }
I20260810 03:50:23.448254 31251 tablet_replica.cc:442] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:50:23.448174 31336 tablet_replica.cc:439] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } } }
I20260810 03:50:23.448297 31336 tablet_replica.cc:442] T 2d166dd7bad349c6981a1513060d77a9 P e33d0fbc1b9c456cb8ce67690ce763b0: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:50:23.450887 31256 tablet_replica.cc:439] T 2d166dd7bad349c6981a1513060d77a9 P 1c2d8d5df0a74479859b89f4fb899221: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c2d8d5df0a74479859b89f4fb899221" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39709 } } peers { permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 } } peers { permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 } } }
I20260810 03:50:23.451022 31256 tablet_replica.cc:442] T 2d166dd7bad349c6981a1513060d77a9 P 1c2d8d5df0a74479859b89f4fb899221: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:50:23.464470 31238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:23.460736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:49759 (local address 127.27.153.67:46553)
0810 03:50:23.460949 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:23.460953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:23.461044 (+ 91us) server_negotiation.cc:407] Connection header received
0810 03:50:23.461184 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:23.461186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:23.461234 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:23.461299 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:50:23.461547 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.462082 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.463308 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.463589 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.463679 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:23.464045 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:50:23.464133 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:50:23.464199 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:23.464298 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 03:50:23.464349 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260810 03:50:23.464546 31345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:23.460606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:46553 (local address 127.27.153.65:49759)
0810 03:50:23.460954 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:23.460971 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:23.461093 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:23.461322 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:23.461324 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:23.461334 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:50:23.461449 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:23.461455 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.462183 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.462186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:23.463164 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:23.463176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.463704 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.463708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:23.463805 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:23.463869 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:50:23.464216 (+ 347us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:50:23.464222 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 03:50:23.464285 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 03:50:23.464385 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":123,"threads_started":1}
I20260810 03:50:23.472084 30853 raft_consensus.cc:1261] T 38e105a87d544b7a938cfbee16c4638a P 97c601af14c240c498822e05c38879a4 [term 1 FOLLOWER]: Refusing update from remote peer 1c2d8d5df0a74479859b89f4fb899221: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:23.472133 31112 raft_consensus.cc:1261] T 38e105a87d544b7a938cfbee16c4638a P e33d0fbc1b9c456cb8ce67690ce763b0 [term 1 FOLLOWER]: Refusing update from remote peer 1c2d8d5df0a74479859b89f4fb899221: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:23.472491 31256 consensus_queue.cc:1035] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e33d0fbc1b9c456cb8ce67690ce763b0" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:23.472682 31247 consensus_queue.cc:1035] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97c601af14c240c498822e05c38879a4" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:23.478580 31266 mvcc.cc:205] Tried to move back new op lower bound from 7316823340938010624 to 7316823339859738624. Current Snapshot: MvccSnapshot[applied={T|T < 7316823340938010624}]
I20260810 03:50:23.498899 31237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:23.495045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:52777 (local address 127.27.153.65:39709)
0810 03:50:23.495162 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:23.495168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:23.495391 (+ 223us) server_negotiation.cc:407] Connection header received
0810 03:50:23.495598 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:23.495601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:23.495654 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:23.495717 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:50:23.496169 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.496788 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.497845 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.498070 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.498168 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:23.498465 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:50:23.498540 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:50:23.498595 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:23.498691 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 03:50:23.498749 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 03:50:23.498914 31355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:23.494975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:39709 (local address 127.27.153.67:52777)
0810 03:50:23.495314 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:23.495327 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:23.495503 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:23.495875 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:23.495878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:23.495886 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:50:23.496026 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:23.496034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.496890 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.496939 (+ 49us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:23.497731 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:23.497741 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.498189 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.498193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:23.498287 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:23.498335 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:50:23.498616 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:50:23.498623 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:50:23.498679 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 03:50:23.498761 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":115,"threads_started":1}
I20260810 03:50:23.525189 30300 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:23.525362 30300 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:23.527575 30300 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e14520d9b09461d9e8faa30a03c8af0: Sending DeleteTablet for 3 replicas of tablet 38e105a87d544b7a938cfbee16c4638a
I20260810 03:50:23.528252 30551 tablet_service.cc:1514] Processing DeleteTablet for tablet 38e105a87d544b7a938cfbee16c4638a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:23 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:39698
I20260810 03:50:23.528254 30818 tablet_service.cc:1514] Processing DeleteTablet for tablet 38e105a87d544b7a938cfbee16c4638a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:23 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:38310
I20260810 03:50:23.528306 31076 tablet_service.cc:1514] Processing DeleteTablet for tablet 38e105a87d544b7a938cfbee16c4638a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:23 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:32808
I20260810 03:50:23.528470 31296 tablet_replica.cc:330] T 38e105a87d544b7a938cfbee16c4638a P e33d0fbc1b9c456cb8ce67690ce763b0: stopping tablet replica
I20260810 03:50:23.528470 31291 tablet_replica.cc:330] T 38e105a87d544b7a938cfbee16c4638a P 97c601af14c240c498822e05c38879a4: stopping tablet replica
I20260810 03:50:23.528470 31293 tablet_replica.cc:330] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221: stopping tablet replica
I20260810 03:50:23.528576 31296 raft_consensus.cc:2229] T 38e105a87d544b7a938cfbee16c4638a P e33d0fbc1b9c456cb8ce67690ce763b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:23.528576 31291 raft_consensus.cc:2229] T 38e105a87d544b7a938cfbee16c4638a P 97c601af14c240c498822e05c38879a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:23.528592 31293 raft_consensus.cc:2229] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:50:23.528698 31296 raft_consensus.cc:2258] T 38e105a87d544b7a938cfbee16c4638a P e33d0fbc1b9c456cb8ce67690ce763b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:23.528698 31291 raft_consensus.cc:2258] T 38e105a87d544b7a938cfbee16c4638a P 97c601af14c240c498822e05c38879a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:23.528818 31293 raft_consensus.cc:2258] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:23.529568 31291 ts_tablet_manager.cc:1905] T 38e105a87d544b7a938cfbee16c4638a P 97c601af14c240c498822e05c38879a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:23.529567 31296 ts_tablet_manager.cc:1905] T 38e105a87d544b7a938cfbee16c4638a P e33d0fbc1b9c456cb8ce67690ce763b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:23.529767 31293 ts_tablet_manager.cc:1905] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:23.532608 31291 ts_tablet_manager.cc:1918] T 38e105a87d544b7a938cfbee16c4638a P 97c601af14c240c498822e05c38879a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:50:23.532691 31291 log.cc:1196] T 38e105a87d544b7a938cfbee16c4638a P 97c601af14c240c498822e05c38879a4: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/38e105a87d544b7a938cfbee16c4638a
I20260810 03:50:23.532944 31291 ts_tablet_manager.cc:1939] T 38e105a87d544b7a938cfbee16c4638a P 97c601af14c240c498822e05c38879a4: Deleting consensus metadata
I20260810 03:50:23.533353 31296 ts_tablet_manager.cc:1918] T 38e105a87d544b7a938cfbee16c4638a P e33d0fbc1b9c456cb8ce67690ce763b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:50:23.533427 31296 log.cc:1196] T 38e105a87d544b7a938cfbee16c4638a P e33d0fbc1b9c456cb8ce67690ce763b0: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/38e105a87d544b7a938cfbee16c4638a
I20260810 03:50:23.533650 31296 ts_tablet_manager.cc:1939] T 38e105a87d544b7a938cfbee16c4638a P e33d0fbc1b9c456cb8ce67690ce763b0: Deleting consensus metadata
I20260810 03:50:23.533784 30278 catalog_manager.cc:4841] TS 97c601af14c240c498822e05c38879a4 (127.27.153.66:34781): tablet 38e105a87d544b7a938cfbee16c4638a (table test-table [id=0047bd2f27bf41f6825fd27d10ab997a]) successfully deleted
I20260810 03:50:23.534349 30284 catalog_manager.cc:4841] TS e33d0fbc1b9c456cb8ce67690ce763b0 (127.27.153.67:46553): tablet 38e105a87d544b7a938cfbee16c4638a (table test-table [id=0047bd2f27bf41f6825fd27d10ab997a]) successfully deleted
I20260810 03:50:23.534840 31293 ts_tablet_manager.cc:1918] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:50:23.534919 31293 log.cc:1196] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786333814837068-28261-0/minicluster-data/ts-0/wal/wals/38e105a87d544b7a938cfbee16c4638a
I20260810 03:50:23.535168 31293 ts_tablet_manager.cc:1939] T 38e105a87d544b7a938cfbee16c4638a P 1c2d8d5df0a74479859b89f4fb899221: Deleting consensus metadata
I20260810 03:50:23.535964 30283 catalog_manager.cc:4841] TS 1c2d8d5df0a74479859b89f4fb899221 (127.27.153.65:39709): tablet 38e105a87d544b7a938cfbee16c4638a (table test-table [id=0047bd2f27bf41f6825fd27d10ab997a]) successfully deleted
Aug 10 03:50:23 dist-test-slave-pww0 krb5kdc[30229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333822, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.153.65@KRBTEST.COM
I20260810 03:50:23.548570 31237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:23.535956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39708 (local address 127.27.153.65:39709)
0810 03:50:23.536072 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:23.536077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:23.536444 (+ 367us) server_negotiation.cc:407] Connection header received
0810 03:50:23.536578 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:23.536580 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:23.536630 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:23.536687 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:23.537606 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.538118 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.539293 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.539546 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.539623 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:23.543742 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:23.543779 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:23.543794 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:23.543829 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:23.545853 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:23.546496 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:23.546503 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:23.546505 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:23.546621 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:23.546947 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:23.546950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:23.546952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:23.547355 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:23.547845 (+ 490us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:23.548254 (+ 409us) server_negotiation.cc:299] Negotiation successful
0810 03:50:23.548384 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 03:50:23.549211 30666 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39708
I20260810 03:50:23.559087 30391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:23.551105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56072 (local address 127.27.153.126:46701)
0810 03:50:23.551235 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:23.551240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:23.551544 (+ 304us) server_negotiation.cc:407] Connection header received
0810 03:50:23.551675 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:23.551677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:23.551724 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:23.551852 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:23.552551 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.553095 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.553880 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:23.554110 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:23.554200 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:23.555630 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:23.555634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:23.555637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:23.555671 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:23.557334 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:23.557789 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:23.557793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:23.557795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:23.557903 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:23.558144 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:23.558147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:23.558149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:23.558413 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:23.558558 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:23.558760 (+ 202us) server_negotiation.cc:299] Negotiation successful
0810 03:50:23.558908 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 03:50:23.559644 30300 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56072
I20260810 03:50:23.561179 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 30435
I20260810 03:50:23.612486 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 30694
I20260810 03:50:23.627192 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 30956
I20260810 03:50:23.637231 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 30249
2026-08-10T03:50:23Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3158 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:50:23 dist-test-slave-pww0 krb5kdc[31368](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:50:23 dist-test-slave-pww0 krb5kdc[31368](info): set up 2 sockets
Aug 10 03:50:23 dist-test-slave-pww0 krb5kdc[31368](info): commencing operation
krb5kdc: starting...
W20260810 03:50:24.844516 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.178s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:50:24 dist-test-slave-pww0 krb5kdc[31368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:50:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:50:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:50:24.978017 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:41135
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:41135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:25.094357 31388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:25.094671 31388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:25.094715 31388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:25.098829 31388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:50:25.098908 31388 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 03:50:25.098928 31388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:25.098948 31388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:25.098966 31388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:50:25.098984 31388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:50:25.104583 31388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:41135
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:41135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:25.105717 31388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:25.106773 31388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:25.115676 31405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:25.115676 31400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:25.115688 31398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:25.115945 31388 server_base.cc:1034] running on GCE node
I20260810 03:50:25.116552 31388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:25.117748 31388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:25.118933 31388 hybrid_clock.cc:648] HybridClock initialized: now 1786333825118904 us; error 42 us; skew 500 ppm
Aug 10 03:50:25 dist-test-slave-pww0 krb5kdc[31368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333825, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:25.124763 31388 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:50:25.125993 31388 webserver.cc:460] Webserver started at http://127.27.153.126:36409/ using document root <none> and password file <none>
I20260810 03:50:25.126729 31388 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:25.126818 31388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:25.127061 31388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:25.130003 31388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "37d0cdd28d594b4eb046ffa44aac8390"
format_stamp: "Formatted at 2026-08-10 03:50:25 on dist-test-slave-pww0"
server_key: "ab86455cbcf354eb3c1d6e060dff66ac"
server_key_iv: "314f5deab9237481c22c0f8bfda10c28"
server_key_version: "encryptionkey@0"
I20260810 03:50:25.130725 31388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "37d0cdd28d594b4eb046ffa44aac8390"
format_stamp: "Formatted at 2026-08-10 03:50:25 on dist-test-slave-pww0"
server_key: "ab86455cbcf354eb3c1d6e060dff66ac"
server_key_iv: "314f5deab9237481c22c0f8bfda10c28"
server_key_version: "encryptionkey@0"
I20260810 03:50:25.134647 31388 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260810 03:50:25.137377 31416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:25.138677 31388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 03:50:25.138792 31388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "37d0cdd28d594b4eb046ffa44aac8390"
format_stamp: "Formatted at 2026-08-10 03:50:25 on dist-test-slave-pww0"
server_key: "ab86455cbcf354eb3c1d6e060dff66ac"
server_key_iv: "314f5deab9237481c22c0f8bfda10c28"
server_key_version: "encryptionkey@0"
I20260810 03:50:25.138916 31388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:25.153545 31388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:25.156883 31388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:25.157071 31388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:25.165671 31388 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:41135
I20260810 03:50:25.165700 31535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:41135 every 8 connection(s)
I20260810 03:50:25.166708 31388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:50:25.169673 31536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:25.175735 31536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390: Bootstrap starting.
I20260810 03:50:25.175778 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 31388
I20260810 03:50:25.175886 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:50:25.176146 28261 external_mini_cluster.cc:1428] Setting key 81ac6f7696d97ec11637442c27d54c86
I20260810 03:50:25.178511 31536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:25.179301 31536 log.cc:824] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:25.181644 31536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390: No bootstrap required, opened a new log
Aug 10 03:50:25 dist-test-slave-pww0 krb5kdc[31368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:25.186195 31536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d0cdd28d594b4eb046ffa44aac8390" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41135 } }
I20260810 03:50:25.186489 31536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:25.186551 31536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37d0cdd28d594b4eb046ffa44aac8390, State: Initialized, Role: FOLLOWER
I20260810 03:50:25.187067 31536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d0cdd28d594b4eb046ffa44aac8390" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41135 } }
I20260810 03:50:25.187259 31536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:50:25.187338 31536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:50:25.187444 31536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:25.188519 31536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d0cdd28d594b4eb046ffa44aac8390" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41135 } }
I20260810 03:50:25.188931 31536 leader_election.cc:302] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37d0cdd28d594b4eb046ffa44aac8390; no voters:
I20260810 03:50:25.189323 31536 leader_election.cc:288] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:50:25.189457 31546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:25.189743 31546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390 [term 1 LEADER]: Becoming Leader. State: Replica: 37d0cdd28d594b4eb046ffa44aac8390, State: Running, Role: LEADER
I20260810 03:50:25.190104 31546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d0cdd28d594b4eb046ffa44aac8390" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41135 } }
I20260810 03:50:25.190744 31536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:50:25.191507 31550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37d0cdd28d594b4eb046ffa44aac8390. Latest consensus state: current_term: 1 leader_uuid: "37d0cdd28d594b4eb046ffa44aac8390" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d0cdd28d594b4eb046ffa44aac8390" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41135 } } }
I20260810 03:50:25.191645 31550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:25.191949 31548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37d0cdd28d594b4eb046ffa44aac8390" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37d0cdd28d594b4eb046ffa44aac8390" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41135 } } }
I20260810 03:50:25.192035 31548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:25.192332 31556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:50:25.193136 31541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:25.177694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49654 (local address 127.27.153.126:41135)
0810 03:50:25.178169 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:25.178185 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:25.178227 (+ 42us) server_negotiation.cc:407] Connection header received
0810 03:50:25.178975 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:25.178996 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:25.179371 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:25.179784 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:25.180579 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.181609 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.182819 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.183355 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.183544 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:25.186501 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:25.186518 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:25.186540 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:25.186584 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:25.188588 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:25.189184 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:25.189192 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:25.189199 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:25.189365 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:25.189774 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:25.189778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:25.189780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:25.190460 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:25.190674 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:25.191537 (+ 863us) server_negotiation.cc:299] Negotiation successful
0810 03:50:25.191917 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":133,"threads_started":1}
I20260810 03:50:25.196195 31556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:50:25.202232 31556 catalog_manager.cc:1347] Generated new cluster ID: f28ca32b7832416c8a543208b383f40c
I20260810 03:50:25.202329 31556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:50:25.219097 31556 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:50:25.220250 31556 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:50:25.228438 31556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390: Generated new TSK 0
I20260810 03:50:25.229348 31556 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.153.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
I20260810 03:50:25.267884 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:41135
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:25.388983 31574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:25.389295 31574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:25.389379 31574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:25.393711 31574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:25.393797 31574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:25.393868 31574 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 03:50:25.393900 31574 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:50:25.393965 31574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:50:25.399497 31574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.153.126:41135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:25.400720 31574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:25.401861 31574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:25.412159 31582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:25.412134 31585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:25.412139 31583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:25.412644 31574 server_base.cc:1034] running on GCE node
I20260810 03:50:25.413087 31574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:25.413759 31574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:25.414953 31574 hybrid_clock.cc:648] HybridClock initialized: now 1786333825414932 us; error 38 us; skew 500 ppm
Aug 10 03:50:25 dist-test-slave-pww0 krb5kdc[31368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333825, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:25.417873 31574 init.cc:377] Logged in from keytab as kudu/127.27.153.65@KRBTEST.COM (short username kudu)
I20260810 03:50:25.418869 31574 webserver.cc:460] Webserver started at http://127.27.153.65:45403/ using document root <none> and password file <none>
I20260810 03:50:25.419423 31574 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:25.419528 31574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:25.419741 31574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:25.421880 31574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "83c344f979094f4f9323e8a15deffa5e"
format_stamp: "Formatted at 2026-08-10 03:50:25 on dist-test-slave-pww0"
server_key: "e1e65f4519bfd386ed7ae80dbb1122ac"
server_key_iv: "1f0aaf99780fad87044ed9042fe74504"
server_key_version: "encryptionkey@0"
I20260810 03:50:25.422452 31574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "83c344f979094f4f9323e8a15deffa5e"
format_stamp: "Formatted at 2026-08-10 03:50:25 on dist-test-slave-pww0"
server_key: "e1e65f4519bfd386ed7ae80dbb1122ac"
server_key_iv: "1f0aaf99780fad87044ed9042fe74504"
server_key_version: "encryptionkey@0"
I20260810 03:50:25.426532 31574 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 03:50:25.429273 31603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:25.430694 31574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:50:25.430825 31574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "83c344f979094f4f9323e8a15deffa5e"
format_stamp: "Formatted at 2026-08-10 03:50:25 on dist-test-slave-pww0"
server_key: "e1e65f4519bfd386ed7ae80dbb1122ac"
server_key_iv: "1f0aaf99780fad87044ed9042fe74504"
server_key_version: "encryptionkey@0"
I20260810 03:50:25.430940 31574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:25.451155 31574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:25.454952 31574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:25.455154 31574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:25.456880 31574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:25.456947 31574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:25.456992 31574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:25.457008 31574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:50:25 dist-test-slave-pww0 krb5kdc[31368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333825, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:25.467641 31574 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:38971
I20260810 03:50:25.467963 31814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:38971 every 8 connection(s)
I20260810 03:50:25.468748 31574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:50:25.471148 31541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:25.457703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:48335 (local address 127.27.153.126:41135)
0810 03:50:25.457894 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:25.457899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:25.458834 (+ 935us) server_negotiation.cc:407] Connection header received
0810 03:50:25.460016 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:25.460020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:25.460082 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:25.460160 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:25.461566 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.462356 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.463787 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.464049 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.464153 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:25.467313 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:25.467318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:25.467321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:25.467358 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:25.468979 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:25.469627 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:25.469631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:25.469632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:25.469734 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:25.470068 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:25.470071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:25.470072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:25.470288 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:25.470383 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:25.470901 (+ 518us) server_negotiation.cc:299] Negotiation successful
0810 03:50:25.471023 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 03:50:25.472083 31623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:25.458126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41135 (local address 127.27.153.65:48335)
0810 03:50:25.458632 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:25.458687 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:25.459757 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:25.460369 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:25.460378 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:25.460756 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:25.461401 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:25.461420 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.462486 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.462489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:25.463657 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:25.463672 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.464237 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.464241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:25.464441 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:25.465166 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:25.465184 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:25.467090 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:25.469111 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:25.469117 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:25.469134 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:25.469508 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:25.469830 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:25.469833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:25.469834 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:25.469975 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:25.470424 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:25.470430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:25.470681 (+ 251us) client_negotiation.cc:769] Sending connection context
0810 03:50:25.470857 (+ 176us) client_negotiation.cc:240] Negotiation successful
0810 03:50:25.471132 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":122,"threads_started":1}
I20260810 03:50:25.473446 31815 heartbeater.cc:347] Connected to a master server at 127.27.153.126:41135
I20260810 03:50:25.473603 31815 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:25.474131 31815 heartbeater.cc:511] Master 127.27.153.126:41135 requested a full tablet report, sending...
I20260810 03:50:25.475853 31443 ts_manager.cc:194] Registered new tserver with Master: 83c344f979094f4f9323e8a15deffa5e (127.27.153.65:38971)
I20260810 03:50:25.476225 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 31574
I20260810 03:50:25.476343 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:50:25.476663 28261 external_mini_cluster.cc:1428] Setting key cbcc756f3395f9acc750c227913b0886
I20260810 03:50:25.477329 31443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:48335
I20260810 03:50:25.483945 31541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:25.475774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:55011 (local address 127.27.153.126:41135)
0810 03:50:25.475900 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:25.475905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:25.476165 (+ 260us) server_negotiation.cc:407] Connection header received
0810 03:50:25.476276 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:25.476279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:25.476321 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:25.476403 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:25.476971 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.477686 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.478752 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.479045 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.479133 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:25.480533 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:25.480536 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:25.480539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:25.480566 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:25.481978 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:25.482470 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:25.482473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:25.482475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:25.482568 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:25.482901 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:25.482904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:25.482905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:25.483168 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:25.483279 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:25.483628 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 03:50:25.483768 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 03:50:25.483968 31822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:25.475688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41135 (local address 127.27.153.65:55011)
0810 03:50:25.476060 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:25.476074 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:25.476196 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:25.476437 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:25.476440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:25.476658 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:25.476835 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:25.476844 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.477812 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.477816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:25.478617 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:25.478629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.479299 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.479302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:25.479393 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:25.479911 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:25.479925 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:25.480392 (+ 467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:25.482130 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:25.482133 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:25.482134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:25.482365 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:25.482700 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:25.482703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:25.482704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:25.482809 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:25.483295 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:25.483297 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:25.483401 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 03:50:25.483563 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 03:50:25.483769 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for kudu/127.27.153.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
I20260810 03:50:25.507409 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:41135
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:25.624305 31828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:25.624594 31828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:25.624643 31828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:25.628720 31828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:25.628794 31828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:25.628836 31828 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 03:50:25.628849 31828 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:50:25.628880 31828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:50:25.634451 31828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.153.126:41135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:25.635735 31828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:25.636879 31828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:25.647313 31837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:25.647323 31835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:25.647308 31842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:25.647667 31828 server_base.cc:1034] running on GCE node
I20260810 03:50:25.648145 31828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:25.648867 31828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:25.650051 31828 hybrid_clock.cc:648] HybridClock initialized: now 1786333825650042 us; error 31 us; skew 500 ppm
Aug 10 03:50:25 dist-test-slave-pww0 krb5kdc[31368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333825, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:25.653195 31828 init.cc:377] Logged in from keytab as kudu/127.27.153.66@KRBTEST.COM (short username kudu)
I20260810 03:50:25.654201 31828 webserver.cc:460] Webserver started at http://127.27.153.66:43229/ using document root <none> and password file <none>
I20260810 03:50:25.654784 31828 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:25.654855 31828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:25.655052 31828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:25.657330 31828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "592678774ca74107a42f086d26a5563a"
format_stamp: "Formatted at 2026-08-10 03:50:25 on dist-test-slave-pww0"
server_key: "06aa6439e946e3b845945224c8a416b0"
server_key_iv: "b01f76b6a3d904089641ad675086d2ce"
server_key_version: "encryptionkey@0"
I20260810 03:50:25.657951 31828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "592678774ca74107a42f086d26a5563a"
format_stamp: "Formatted at 2026-08-10 03:50:25 on dist-test-slave-pww0"
server_key: "06aa6439e946e3b845945224c8a416b0"
server_key_iv: "b01f76b6a3d904089641ad675086d2ce"
server_key_version: "encryptionkey@0"
I20260810 03:50:25.662215 31828 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260810 03:50:25.664983 31854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:25.666332 31828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:50:25.666461 31828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "592678774ca74107a42f086d26a5563a"
format_stamp: "Formatted at 2026-08-10 03:50:25 on dist-test-slave-pww0"
server_key: "06aa6439e946e3b845945224c8a416b0"
server_key_iv: "b01f76b6a3d904089641ad675086d2ce"
server_key_version: "encryptionkey@0"
I20260810 03:50:25.666579 31828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:25.685199 31828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:25.688542 31828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:25.688738 31828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:25.690335 31828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:25.690394 31828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:25.690451 31828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:25.690478 31828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:50:25 dist-test-slave-pww0 krb5kdc[31368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333825, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:25.701716 31828 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:43007
I20260810 03:50:25.701784 32040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:43007 every 8 connection(s)
I20260810 03:50:25.702811 31828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:50:25.704329 31541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:25.691199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:39937 (local address 127.27.153.126:41135)
0810 03:50:25.691370 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:25.691375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:25.692510 (+ 1135us) server_negotiation.cc:407] Connection header received
0810 03:50:25.693775 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:25.693779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:25.693847 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:25.693952 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:25.695347 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.695939 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.697303 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.697608 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.697740 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:25.700501 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:25.700505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:25.700508 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:25.700540 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:25.702143 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:25.702640 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:25.702643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:25.702645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:25.702745 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:25.703115 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:25.703119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:25.703120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:25.703371 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:25.703505 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:25.704082 (+ 577us) server_negotiation.cc:299] Negotiation successful
0810 03:50:25.704203 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:50:25.705153 31870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:25.691730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41135 (local address 127.27.153.66:39937)
0810 03:50:25.692311 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:25.692377 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:25.693499 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:25.694168 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:25.694178 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:25.694547 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:25.695096 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:25.695112 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.696070 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.696073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:25.697178 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:25.697191 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.697784 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.697788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:25.697991 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:25.698667 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:25.698690 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:25.700310 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:25.702263 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:25.702268 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:25.702280 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:25.702533 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:25.702840 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:25.702842 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:25.702843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:25.703018 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:25.703562 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:25.703569 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:25.703849 (+ 280us) client_negotiation.cc:769] Sending connection context
0810 03:50:25.704013 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 03:50:25.704283 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":148,"threads_started":1}
I20260810 03:50:25.705588 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 31828
I20260810 03:50:25.705689 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:50:25.705982 28261 external_mini_cluster.cc:1428] Setting key 2c804e13c36cc9926fbe780ee28e3c9a
I20260810 03:50:25.706471 32041 heartbeater.cc:347] Connected to a master server at 127.27.153.126:41135
I20260810 03:50:25.706629 32041 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:25.707212 32041 heartbeater.cc:511] Master 127.27.153.126:41135 requested a full tablet report, sending...
I20260810 03:50:25.708390 31444 ts_manager.cc:194] Registered new tserver with Master: 592678774ca74107a42f086d26a5563a (127.27.153.66:43007)
I20260810 03:50:25.709134 31444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:39937
WARNING: no policy specified for kudu/127.27.153.67@KRBTEST.COM; defaulting to no policy
I20260810 03:50:25.716872 31541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:25.708837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:43083 (local address 127.27.153.126:41135)
0810 03:50:25.708946 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:25.708950 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:25.709213 (+ 263us) server_negotiation.cc:407] Connection header received
0810 03:50:25.709324 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:25.709327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:25.709377 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:25.709455 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:25.710088 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.710851 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.711924 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.712159 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.712273 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:25.713665 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:25.713668 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:25.713670 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:25.713697 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:25.715106 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:25.715596 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:25.715599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:25.715600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:25.715714 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:25.716003 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:25.716005 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:25.716006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:25.716239 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:25.716317 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:25.716614 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 03:50:25.716748 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 03:50:25.716899 32055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:25.708755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41135 (local address 127.27.153.66:43083)
0810 03:50:25.709108 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:25.709122 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:25.709238 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:25.709583 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:25.709587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:25.709817 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:25.709950 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:25.709958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.711015 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.711019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:25.711808 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:25.711818 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.712313 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.712317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:25.712424 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:25.712954 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:25.712969 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:25.713437 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:25.715223 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:25.715227 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:25.715230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:25.715502 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:25.715810 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:25.715813 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:25.715815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:25.715913 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:25.716329 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:25.716332 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:25.716443 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 03:50:25.716553 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 03:50:25.716721 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":1,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
I20260810 03:50:25.734861 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:41135
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:33725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:25.859105 32065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:25.859423 32065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:25.859524 32065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:25.863680 32065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:25.863749 32065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:25.863807 32065 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 03:50:25.863834 32065 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:50:25.863884 32065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:50:25.869601 32065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:33725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.153.126:41135
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:25.870918 32065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:25.872030 32065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:25.882263 32074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:25.882346 32077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:25.882447 32065 server_base.cc:1034] running on GCE node
W20260810 03:50:25.882678 32073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:25.883116 32065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:25.883889 32065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:25.885095 32065 hybrid_clock.cc:648] HybridClock initialized: now 1786333825885051 us; error 67 us; skew 500 ppm
Aug 10 03:50:25 dist-test-slave-pww0 krb5kdc[31368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333825, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:25.888136 32065 init.cc:377] Logged in from keytab as kudu/127.27.153.67@KRBTEST.COM (short username kudu)
I20260810 03:50:25.889273 32065 webserver.cc:460] Webserver started at http://127.27.153.67:40739/ using document root <none> and password file <none>
I20260810 03:50:25.889904 32065 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:25.889977 32065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:25.890204 32065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:25.892881 32065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "f2d027e0ea9c4e96bbbdfb676999a27b"
format_stamp: "Formatted at 2026-08-10 03:50:25 on dist-test-slave-pww0"
server_key: "1fb1d26849b276b097d6d3b61f279c9a"
server_key_iv: "bcd34a01f8db60fe3a9efecbab5f1bef"
server_key_version: "encryptionkey@0"
I20260810 03:50:25.893539 32065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "f2d027e0ea9c4e96bbbdfb676999a27b"
format_stamp: "Formatted at 2026-08-10 03:50:25 on dist-test-slave-pww0"
server_key: "1fb1d26849b276b097d6d3b61f279c9a"
server_key_iv: "bcd34a01f8db60fe3a9efecbab5f1bef"
server_key_version: "encryptionkey@0"
I20260810 03:50:25.898092 32065 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:50:25.900899 32088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:25.902346 32065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:50:25.902482 32065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "f2d027e0ea9c4e96bbbdfb676999a27b"
format_stamp: "Formatted at 2026-08-10 03:50:25 on dist-test-slave-pww0"
server_key: "1fb1d26849b276b097d6d3b61f279c9a"
server_key_iv: "bcd34a01f8db60fe3a9efecbab5f1bef"
server_key_version: "encryptionkey@0"
I20260810 03:50:25.902630 32065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:25.920068 32065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:25.923823 32065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:25.924039 32065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:25.925786 32065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:25.925849 32065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:25.925896 32065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:25.925926 32065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:50:25 dist-test-slave-pww0 krb5kdc[31368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333825, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:25.936937 32065 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:37723
I20260810 03:50:25.937038 32278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:37723 every 8 connection(s)
I20260810 03:50:25.938014 32065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:50:25.941679 31541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:25.927153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:60265 (local address 127.27.153.126:41135)
0810 03:50:25.927294 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:25.927299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:25.928567 (+ 1268us) server_negotiation.cc:407] Connection header received
0810 03:50:25.929879 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:25.929883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:25.929948 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:25.930025 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:25.931450 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.932311 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.933983 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.934259 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.934367 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:25.937828 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:25.937833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:25.937836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:25.937872 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:25.939521 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:25.940068 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:25.940072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:25.940074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:25.940164 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:25.940498 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:25.940501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:25.940503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:25.940720 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:25.940811 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:25.941356 (+ 545us) server_negotiation.cc:299] Negotiation successful
0810 03:50:25.941495 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:50:25.942610 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:25.927786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41135 (local address 127.27.153.67:60265)
0810 03:50:25.928328 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:25.928386 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:25.929625 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:25.930231 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:25.930240 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:25.930746 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:25.931280 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:25.931295 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.932441 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.932445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:25.933722 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:25.933734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.934433 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.934437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:25.934670 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:25.935405 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:25.935431 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:25.937510 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:25.939654 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:25.939660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:25.939678 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:25.939960 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:25.940252 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:25.940255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:25.940257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:25.940407 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:25.940824 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:25.940832 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:25.941106 (+ 274us) client_negotiation.cc:769] Sending connection context
0810 03:50:25.941285 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 03:50:25.941653 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":141,"threads_started":1}
I20260810 03:50:25.943579 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 32065
I20260810 03:50:25.943742 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:50:25.944036 32279 heartbeater.cc:347] Connected to a master server at 127.27.153.126:41135
I20260810 03:50:25.944096 28261 external_mini_cluster.cc:1428] Setting key 359bf84263985c9abdfcf99c350db6b0
I20260810 03:50:25.944217 32279 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:25.944868 32279 heartbeater.cc:511] Master 127.27.153.126:41135 requested a full tablet report, sending...
I20260810 03:50:25.945991 31444 ts_manager.cc:194] Registered new tserver with Master: f2d027e0ea9c4e96bbbdfb676999a27b (127.27.153.67:37723)
I20260810 03:50:25.946678 31444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:60265
I20260810 03:50:25.954891 32294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:25.947148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41135 (local address 127.27.153.67:45355)
0810 03:50:25.947507 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:25.947523 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:25.947641 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:25.947926 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:25.947928 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:25.948168 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:25.948291 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:25.948299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.949260 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.949263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:25.949883 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:25.949893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.950381 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.950384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:25.950457 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:25.950969 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:25.950982 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:25.951348 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:25.953201 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:25.953204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:25.953205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:25.953416 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:25.953797 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:25.953800 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:25.953802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:25.953897 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:25.954347 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:25.954350 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:25.954467 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 03:50:25.954568 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 03:50:25.954706 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":121,"threads_started":1}
I20260810 03:50:25.954938 31541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:25.947214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:45355 (local address 127.27.153.126:41135)
0810 03:50:25.947349 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:25.947354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:25.947590 (+ 236us) server_negotiation.cc:407] Connection header received
0810 03:50:25.947730 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:25.947732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:25.947818 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:25.947887 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:25.948450 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.949093 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.950009 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.950257 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.950360 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:25.951544 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:25.951548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:25.951551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:25.951579 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:25.953085 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:25.953538 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:25.953541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:25.953543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:25.953661 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:25.953982 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:25.953985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:25.953986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:25.954195 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:25.954284 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:25.954675 (+ 391us) server_negotiation.cc:299] Negotiation successful
0810 03:50:25.954812 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 03:50:25.958418 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 03:50:25 dist-test-slave-pww0 krb5kdc[31368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 03:50:25 dist-test-slave-pww0 krb5kdc[31368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:25.987787 31541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:25.978818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49662 (local address 127.27.153.126:41135)
0810 03:50:25.978958 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:25.978961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:25.978975 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:50:25.979020 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:25.979022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:25.979073 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:25.979156 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:25.980019 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.980579 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.981455 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:25.981688 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:25.981788 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:25.984669 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:25.984672 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:25.984674 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:25.984706 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:25.986002 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:25.986430 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:25.986434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:25.986435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:25.986516 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:25.986858 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:25.986862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:25.986880 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:25.987076 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:25.987198 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:25.987500 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 03:50:25.987625 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:50:25.990972 31444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:50:25.994282 31444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:50:26.010229 32321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.004503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59560 (local address 127.27.153.66:43007)
0810 03:50:26.005068 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:26.005080 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:26.005425 (+ 345us) server_negotiation.cc:407] Connection header received
0810 03:50:26.005510 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:26.005515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:26.005702 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:26.005898 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:26.006039 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.007126 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.008748 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.009473 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.009587 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.009786 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:26.009950 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 03:50:26.010058 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":116,"threads_started":1}
I20260810 03:50:26.010231 32314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.003701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:38971 (local address 127.0.0.1:46752)
0810 03:50:26.004400 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:26.004414 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:26.004574 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:26.005325 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:26.005332 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:26.005358 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:26.005539 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.005547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.006981 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.006984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.008479 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.008491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.009870 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.009874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.009974 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.009995 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 03:50:26.010056 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 03:50:26.010109 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":3,"thread_start_us":119,"threads_started":1}
I20260810 03:50:26.010231 32320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.004319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:43007 (local address 127.0.0.1:59560)
0810 03:50:26.005212 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:26.005225 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:26.005314 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:26.005869 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:26.005871 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:26.005891 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:26.005960 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.005964 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.007334 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.007338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.008608 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.008616 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.009598 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.009602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.009743 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.009788 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 03:50:26.009909 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 03:50:26.010066 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":101,"threads_started":1}
I20260810 03:50:26.011171 32312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.003425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:37723 (local address 127.0.0.1:60278)
0810 03:50:26.003898 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:26.003959 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:26.004096 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:26.005392 (+ 1296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:26.005394 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:26.005408 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:26.005539 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.005547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.008234 (+ 2687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.008237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.009326 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.009339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.010722 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.010726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.010832 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.010857 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 03:50:26.010929 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 03:50:26.011003 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":37,"thread_start_us":151,"threads_started":1}
I20260810 03:50:26.011324 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.004254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60278 (local address 127.27.153.67:37723)
0810 03:50:26.004855 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:26.004867 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:26.004887 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:50:26.004956 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:26.004961 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:26.005122 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:26.005259 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:26.007022 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.008095 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.009472 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.010250 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.010469 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.010637 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:26.010995 (+ 358us) server_negotiation.cc:299] Negotiation successful
0810 03:50:26.011098 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":141,"threads_started":1}
I20260810 03:50:26.011601 32316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.003852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46752 (local address 127.27.153.65:38971)
0810 03:50:26.004253 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:26.004261 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:26.004505 (+ 244us) server_negotiation.cc:407] Connection header received
0810 03:50:26.004725 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:26.004731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:26.004956 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:26.005104 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:26.005664 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.006868 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.008626 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.009552 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.009652 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.009808 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:26.010753 (+ 945us) server_negotiation.cc:299] Negotiation successful
0810 03:50:26.010848 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":76,"threads_started":1}
I20260810 03:50:26.014175 31695 tablet_service.cc:1467] Processing CreateTablet for tablet 87ea02d8f6fc4a30ad9fd54f7c228914 (DEFAULT_TABLE table=test-table [id=e4266eb33f5048f0b61532932b5b14e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:26.014209 32175 tablet_service.cc:1467] Processing CreateTablet for tablet 87ea02d8f6fc4a30ad9fd54f7c228914 (DEFAULT_TABLE table=test-table [id=e4266eb33f5048f0b61532932b5b14e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:26.014209 31942 tablet_service.cc:1467] Processing CreateTablet for tablet 87ea02d8f6fc4a30ad9fd54f7c228914 (DEFAULT_TABLE table=test-table [id=e4266eb33f5048f0b61532932b5b14e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:26.015190 31695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87ea02d8f6fc4a30ad9fd54f7c228914. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:26.015276 31942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87ea02d8f6fc4a30ad9fd54f7c228914. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:26.015291 32175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87ea02d8f6fc4a30ad9fd54f7c228914. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:26.022258 32322 tablet_bootstrap.cc:492] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 83c344f979094f4f9323e8a15deffa5e: Bootstrap starting.
I20260810 03:50:26.023895 32324 tablet_bootstrap.cc:492] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 592678774ca74107a42f086d26a5563a: Bootstrap starting.
I20260810 03:50:26.024704 32322 tablet_bootstrap.cc:654] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 83c344f979094f4f9323e8a15deffa5e: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:26.025774 32324 tablet_bootstrap.cc:654] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 592678774ca74107a42f086d26a5563a: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:26.025825 32322 log.cc:824] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 83c344f979094f4f9323e8a15deffa5e: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:26.026333 32323 tablet_bootstrap.cc:492] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b: Bootstrap starting.
I20260810 03:50:26.026633 32324 log.cc:824] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 592678774ca74107a42f086d26a5563a: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:26.028082 32322 tablet_bootstrap.cc:492] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 83c344f979094f4f9323e8a15deffa5e: No bootstrap required, opened a new log
I20260810 03:50:26.028291 32322 ts_tablet_manager.cc:1397] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 83c344f979094f4f9323e8a15deffa5e: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:50:26.028666 32323 tablet_bootstrap.cc:654] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:26.028774 32324 tablet_bootstrap.cc:492] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 592678774ca74107a42f086d26a5563a: No bootstrap required, opened a new log
I20260810 03:50:26.028968 32324 ts_tablet_manager.cc:1397] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 592678774ca74107a42f086d26a5563a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 03:50:26.029562 32323 log.cc:824] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:26.031807 32323 tablet_bootstrap.cc:492] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b: No bootstrap required, opened a new log
I20260810 03:50:26.031687 32322 raft_consensus.cc:348] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } } peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } }
I20260810 03:50:26.031947 32322 raft_consensus.cc:374] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:26.032001 32322 raft_consensus.cc:729] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c344f979094f4f9323e8a15deffa5e, State: Initialized, Role: FOLLOWER
I20260810 03:50:26.032015 32323 ts_tablet_manager.cc:1397] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:50:26.032441 32322 consensus_queue.cc:260] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 83c344f979094f4f9323e8a15deffa5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } } peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } }
I20260810 03:50:26.032776 32324 raft_consensus.cc:348] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } } peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } }
I20260810 03:50:26.033046 32324 raft_consensus.cc:374] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:26.033095 32324 raft_consensus.cc:729] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 592678774ca74107a42f086d26a5563a, State: Initialized, Role: FOLLOWER
I20260810 03:50:26.033087 31815 heartbeater.cc:503] Master 127.27.153.126:41135 was elected leader, sending a full tablet report...
I20260810 03:50:26.033460 32322 ts_tablet_manager.cc:1428] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 83c344f979094f4f9323e8a15deffa5e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 03:50:26.033578 32324 consensus_queue.cc:260] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 592678774ca74107a42f086d26a5563a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } } peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } }
I20260810 03:50:26.034304 32041 heartbeater.cc:503] Master 127.27.153.126:41135 was elected leader, sending a full tablet report...
I20260810 03:50:26.035802 32324 ts_tablet_manager.cc:1428] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 592678774ca74107a42f086d26a5563a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 03:50:26.036145 32323 raft_consensus.cc:348] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } } peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } }
I20260810 03:50:26.036367 32323 raft_consensus.cc:374] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:26.036399 32323 raft_consensus.cc:729] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2d027e0ea9c4e96bbbdfb676999a27b, State: Initialized, Role: FOLLOWER
I20260810 03:50:26.036842 32323 consensus_queue.cc:260] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } } peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } }
I20260810 03:50:26.037598 32279 heartbeater.cc:503] Master 127.27.153.126:41135 was elected leader, sending a full tablet report...
I20260810 03:50:26.037989 32323 ts_tablet_manager.cc:1428] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 03:50:26.075132 32333 raft_consensus.cc:482] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:50:26.075376 32333 raft_consensus.cc:504] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } } peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } }
I20260810 03:50:26.076701 32333 leader_election.cc:288] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83c344f979094f4f9323e8a15deffa5e (127.27.153.65:38971), 592678774ca74107a42f086d26a5563a (127.27.153.66:43007)
I20260810 03:50:26.080837 32101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.077009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:43007 (local address 127.27.153.67:33539)
0810 03:50:26.077135 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:26.077153 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:26.077274 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:26.077545 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:26.077548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:26.077570 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:26.077688 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.077698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.078543 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.078547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.079623 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.079635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.080318 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.080323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.080445 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.080477 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 03:50:26.080585 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 03:50:26.080675 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 03:50:26.080832 32321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.077189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:33539 (local address 127.27.153.66:43007)
0810 03:50:26.077307 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:26.077310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:26.077325 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:50:26.077379 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:26.077381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:26.077422 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:26.077513 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:26.077808 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.078414 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.079759 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.080188 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.080299 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.080346 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:26.080626 (+ 280us) server_negotiation.cc:299] Negotiation successful
0810 03:50:26.080698 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 03:50:26.081864 32316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.077110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:42305 (local address 127.27.153.65:38971)
0810 03:50:26.077263 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:26.077268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:26.077923 (+ 655us) server_negotiation.cc:407] Connection header received
0810 03:50:26.078067 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:26.078069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:26.078138 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:26.078210 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:26.079123 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.079669 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.080842 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.081326 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.081445 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.081498 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:26.081671 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 03:50:26.081752 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 03:50:26.081892 32334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.077009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:38971 (local address 127.27.153.67:42305)
0810 03:50:26.077815 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:26.077829 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:26.077944 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:26.078234 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:26.078236 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:26.078256 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:26.078973 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.078988 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.079802 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.079804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.080698 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.080709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.081464 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.081468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.081563 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.081581 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 03:50:26.081648 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 03:50:26.081732 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":334,"thread_start_us":154,"threads_started":1}
I20260810 03:50:26.081815 31976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87ea02d8f6fc4a30ad9fd54f7c228914" candidate_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "592678774ca74107a42f086d26a5563a" is_pre_election: true
I20260810 03:50:26.082203 31976 raft_consensus.cc:2454] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2d027e0ea9c4e96bbbdfb676999a27b in term 0.
I20260810 03:50:26.082573 31732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87ea02d8f6fc4a30ad9fd54f7c228914" candidate_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c344f979094f4f9323e8a15deffa5e" is_pre_election: true
I20260810 03:50:26.082823 32093 leader_election.cc:302] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 592678774ca74107a42f086d26a5563a, f2d027e0ea9c4e96bbbdfb676999a27b; no voters:
I20260810 03:50:26.082887 31732 raft_consensus.cc:2454] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2d027e0ea9c4e96bbbdfb676999a27b in term 0.
I20260810 03:50:26.083299 32333 raft_consensus.cc:2781] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:50:26.083408 32333 raft_consensus.cc:482] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:50:26.083489 32333 raft_consensus.cc:3037] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:26.084625 32333 raft_consensus.cc:504] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } } peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } }
I20260810 03:50:26.084995 32333 leader_election.cc:288] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b [CANDIDATE]: Term 1 election: Requested vote from peers 83c344f979094f4f9323e8a15deffa5e (127.27.153.65:38971), 592678774ca74107a42f086d26a5563a (127.27.153.66:43007)
I20260810 03:50:26.085477 31732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87ea02d8f6fc4a30ad9fd54f7c228914" candidate_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c344f979094f4f9323e8a15deffa5e"
I20260810 03:50:26.085597 31976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87ea02d8f6fc4a30ad9fd54f7c228914" candidate_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "592678774ca74107a42f086d26a5563a"
I20260810 03:50:26.085657 31732 raft_consensus.cc:3037] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:26.085736 31976 raft_consensus.cc:3037] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:26.087056 31732 raft_consensus.cc:2454] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 83c344f979094f4f9323e8a15deffa5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2d027e0ea9c4e96bbbdfb676999a27b in term 1.
I20260810 03:50:26.087056 31976 raft_consensus.cc:2454] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 592678774ca74107a42f086d26a5563a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2d027e0ea9c4e96bbbdfb676999a27b in term 1.
I20260810 03:50:26.087440 32092 leader_election.cc:302] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83c344f979094f4f9323e8a15deffa5e, f2d027e0ea9c4e96bbbdfb676999a27b; no voters:
I20260810 03:50:26.087723 32333 raft_consensus.cc:2781] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:26.087973 32333 raft_consensus.cc:686] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b [term 1 LEADER]: Becoming Leader. State: Replica: f2d027e0ea9c4e96bbbdfb676999a27b, State: Running, Role: LEADER
I20260810 03:50:26.088312 32333 consensus_queue.cc:237] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } } peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } }
I20260810 03:50:26.091918 31444 catalog_manager.cc:5495] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b reported cstate change: term changed from 0 to 1, leader changed from <none> to f2d027e0ea9c4e96bbbdfb676999a27b (127.27.153.67). New cstate: current_term: 1 leader_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } health_report { overall_health: UNKNOWN } } }
Aug 10 03:50:26 dist-test-slave-pww0 krb5kdc[31368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333825, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:26.146234 31541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.137520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49670 (local address 127.27.153.126:41135)
0810 03:50:26.137662 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:26.137666 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:26.137899 (+ 233us) server_negotiation.cc:407] Connection header received
0810 03:50:26.138006 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:26.138008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:26.138053 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:26.138140 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:26.138772 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.139303 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.140207 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.140441 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.140524 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.142952 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:26.142957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:26.142960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:26.142984 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:26.144604 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:26.145122 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:26.145124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:26.145125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:26.145194 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:26.145427 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:26.145429 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:26.145430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:26.145599 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:26.145669 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:26.145934 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 03:50:26.146085 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":54}
I20260810 03:50:26.146235 32349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.137447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41135 (local address 127.0.0.1:49670)
0810 03:50:26.137808 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:26.137821 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:26.137934 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:26.138162 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:26.138164 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:26.138562 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:26.138665 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.138672 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.139404 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.139407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.140092 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.140103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.140537 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.140539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.140600 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.141169 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:26.141184 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:26.142802 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:26.144739 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:26.144753 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:26.144761 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:26.145034 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:26.145267 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:26.145270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:26.145271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:26.145362 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:26.145679 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:26.145686 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:26.145801 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 03:50:26.145892 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 03:50:26.146029 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":177,"threads_started":1}
I20260810 03:50:26.150538 31444 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:49670:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 03:50:26.151347 31444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:50:26.156421 31695 tablet_service.cc:1467] Processing CreateTablet for tablet e80fca4f6b73409dbd3bc2085bc3dee6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=390fd85805fa42f28d5d4bc123ff76f1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:50:26.156575 31942 tablet_service.cc:1467] Processing CreateTablet for tablet e80fca4f6b73409dbd3bc2085bc3dee6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=390fd85805fa42f28d5d4bc123ff76f1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:50:26.156610 32175 tablet_service.cc:1467] Processing CreateTablet for tablet e80fca4f6b73409dbd3bc2085bc3dee6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=390fd85805fa42f28d5d4bc123ff76f1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:50:26.157244 31695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e80fca4f6b73409dbd3bc2085bc3dee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:26.157243 31942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e80fca4f6b73409dbd3bc2085bc3dee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:26.157325 32175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e80fca4f6b73409dbd3bc2085bc3dee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:26.159854 32323 tablet_bootstrap.cc:492] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b: Bootstrap starting.
I20260810 03:50:26.160226 32324 tablet_bootstrap.cc:492] T e80fca4f6b73409dbd3bc2085bc3dee6 P 592678774ca74107a42f086d26a5563a: Bootstrap starting.
I20260810 03:50:26.160900 32322 tablet_bootstrap.cc:492] T e80fca4f6b73409dbd3bc2085bc3dee6 P 83c344f979094f4f9323e8a15deffa5e: Bootstrap starting.
I20260810 03:50:26.160934 32323 tablet_bootstrap.cc:654] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:26.161216 32324 tablet_bootstrap.cc:654] T e80fca4f6b73409dbd3bc2085bc3dee6 P 592678774ca74107a42f086d26a5563a: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:26.161827 32322 tablet_bootstrap.cc:654] T e80fca4f6b73409dbd3bc2085bc3dee6 P 83c344f979094f4f9323e8a15deffa5e: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:26.162287 32323 tablet_bootstrap.cc:492] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b: No bootstrap required, opened a new log
I20260810 03:50:26.162379 32323 ts_tablet_manager.cc:1397] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:50:26.162700 32324 tablet_bootstrap.cc:492] T e80fca4f6b73409dbd3bc2085bc3dee6 P 592678774ca74107a42f086d26a5563a: No bootstrap required, opened a new log
I20260810 03:50:26.162781 32324 ts_tablet_manager.cc:1397] T e80fca4f6b73409dbd3bc2085bc3dee6 P 592678774ca74107a42f086d26a5563a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:50:26.162796 32323 raft_consensus.cc:348] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:26.162909 32323 raft_consensus.cc:374] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:26.162940 32323 raft_consensus.cc:729] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2d027e0ea9c4e96bbbdfb676999a27b, State: Initialized, Role: FOLLOWER
I20260810 03:50:26.163074 32323 consensus_queue.cc:260] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:26.163187 32322 tablet_bootstrap.cc:492] T e80fca4f6b73409dbd3bc2085bc3dee6 P 83c344f979094f4f9323e8a15deffa5e: No bootstrap required, opened a new log
I20260810 03:50:26.163173 32324 raft_consensus.cc:348] T e80fca4f6b73409dbd3bc2085bc3dee6 P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:26.163267 32322 ts_tablet_manager.cc:1397] T e80fca4f6b73409dbd3bc2085bc3dee6 P 83c344f979094f4f9323e8a15deffa5e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 03:50:26.163273 32324 raft_consensus.cc:374] T e80fca4f6b73409dbd3bc2085bc3dee6 P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:26.163304 32324 raft_consensus.cc:729] T e80fca4f6b73409dbd3bc2085bc3dee6 P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 592678774ca74107a42f086d26a5563a, State: Initialized, Role: FOLLOWER
I20260810 03:50:26.163345 32323 ts_tablet_manager.cc:1428] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 03:50:26.163417 32324 consensus_queue.cc:260] T e80fca4f6b73409dbd3bc2085bc3dee6 P 592678774ca74107a42f086d26a5563a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:26.163690 32324 ts_tablet_manager.cc:1428] T e80fca4f6b73409dbd3bc2085bc3dee6 P 592678774ca74107a42f086d26a5563a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:50:26.163707 32322 raft_consensus.cc:348] T e80fca4f6b73409dbd3bc2085bc3dee6 P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:26.163818 32322 raft_consensus.cc:374] T e80fca4f6b73409dbd3bc2085bc3dee6 P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:26.163856 32322 raft_consensus.cc:729] T e80fca4f6b73409dbd3bc2085bc3dee6 P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c344f979094f4f9323e8a15deffa5e, State: Initialized, Role: FOLLOWER
I20260810 03:50:26.163901 32332 tablet_replica.cc:439] T e80fca4f6b73409dbd3bc2085bc3dee6 P 592678774ca74107a42f086d26a5563a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } } }
I20260810 03:50:26.163980 32322 consensus_queue.cc:260] T e80fca4f6b73409dbd3bc2085bc3dee6 P 83c344f979094f4f9323e8a15deffa5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:26.164043 32332 tablet_replica.cc:442] T e80fca4f6b73409dbd3bc2085bc3dee6 P 592678774ca74107a42f086d26a5563a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:50:26.164211 32322 ts_tablet_manager.cc:1428] T e80fca4f6b73409dbd3bc2085bc3dee6 P 83c344f979094f4f9323e8a15deffa5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:50:26.164253 32331 tablet_replica.cc:439] T e80fca4f6b73409dbd3bc2085bc3dee6 P 83c344f979094f4f9323e8a15deffa5e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } } }
I20260810 03:50:26.164395 32331 tablet_replica.cc:442] T e80fca4f6b73409dbd3bc2085bc3dee6 P 83c344f979094f4f9323e8a15deffa5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:50:26.163342 32333 tablet_replica.cc:439] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } } }
I20260810 03:50:26.164938 32333 tablet_replica.cc:442] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b: This TxnStatusTablet replica's current role is: FOLLOWER
W20260810 03:50:26.189484 32284 tablet.cc:2369] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:50:26.204144 32043 tablet.cc:2369] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 592678774ca74107a42f086d26a5563a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:50:26.220623 31816 tablet.cc:2369] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 83c344f979094f4f9323e8a15deffa5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:50:26.245546 32333 raft_consensus.cc:482] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:50:26.245711 32333 raft_consensus.cc:504] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:26.246178 32333 leader_election.cc:288] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83c344f979094f4f9323e8a15deffa5e (127.27.153.65:38971), 592678774ca74107a42f086d26a5563a (127.27.153.66:43007)
I20260810 03:50:26.246783 31732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e80fca4f6b73409dbd3bc2085bc3dee6" candidate_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c344f979094f4f9323e8a15deffa5e" is_pre_election: true
I20260810 03:50:26.246785 31976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e80fca4f6b73409dbd3bc2085bc3dee6" candidate_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "592678774ca74107a42f086d26a5563a" is_pre_election: true
I20260810 03:50:26.246935 31732 raft_consensus.cc:2454] T e80fca4f6b73409dbd3bc2085bc3dee6 P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2d027e0ea9c4e96bbbdfb676999a27b in term 0.
I20260810 03:50:26.246943 31976 raft_consensus.cc:2454] T e80fca4f6b73409dbd3bc2085bc3dee6 P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2d027e0ea9c4e96bbbdfb676999a27b in term 0.
I20260810 03:50:26.247265 32093 leader_election.cc:302] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 592678774ca74107a42f086d26a5563a, f2d027e0ea9c4e96bbbdfb676999a27b; no voters:
I20260810 03:50:26.247506 32333 raft_consensus.cc:2781] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:50:26.247577 32333 raft_consensus.cc:482] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:50:26.247604 32333 raft_consensus.cc:3037] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:26.248605 32333 raft_consensus.cc:504] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:26.248978 32333 leader_election.cc:288] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b [CANDIDATE]: Term 1 election: Requested vote from peers 83c344f979094f4f9323e8a15deffa5e (127.27.153.65:38971), 592678774ca74107a42f086d26a5563a (127.27.153.66:43007)
I20260810 03:50:26.249425 31732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e80fca4f6b73409dbd3bc2085bc3dee6" candidate_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c344f979094f4f9323e8a15deffa5e"
I20260810 03:50:26.249498 31976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e80fca4f6b73409dbd3bc2085bc3dee6" candidate_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "592678774ca74107a42f086d26a5563a"
I20260810 03:50:26.249598 31732 raft_consensus.cc:3037] T e80fca4f6b73409dbd3bc2085bc3dee6 P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:26.249640 31976 raft_consensus.cc:3037] T e80fca4f6b73409dbd3bc2085bc3dee6 P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:26.250588 31976 raft_consensus.cc:2454] T e80fca4f6b73409dbd3bc2085bc3dee6 P 592678774ca74107a42f086d26a5563a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2d027e0ea9c4e96bbbdfb676999a27b in term 1.
I20260810 03:50:26.250588 31732 raft_consensus.cc:2454] T e80fca4f6b73409dbd3bc2085bc3dee6 P 83c344f979094f4f9323e8a15deffa5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2d027e0ea9c4e96bbbdfb676999a27b in term 1.
I20260810 03:50:26.250918 32093 leader_election.cc:302] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 592678774ca74107a42f086d26a5563a, f2d027e0ea9c4e96bbbdfb676999a27b; no voters:
I20260810 03:50:26.251119 32333 raft_consensus.cc:2781] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:26.251216 32333 raft_consensus.cc:686] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b [term 1 LEADER]: Becoming Leader. State: Replica: f2d027e0ea9c4e96bbbdfb676999a27b, State: Running, Role: LEADER
I20260810 03:50:26.251363 32333 consensus_queue.cc:237] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:26.252095 32336 tablet_replica.cc:439] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b: TxnStatusTablet state changed. Reason: New leader f2d027e0ea9c4e96bbbdfb676999a27b. Latest consensus state: current_term: 1 leader_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } } }
I20260810 03:50:26.252233 32336 tablet_replica.cc:442] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:50:26.252550 32363 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 03:50:26.252799 32363 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 03:50:26.253314 31440 catalog_manager.cc:5495] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b reported cstate change: term changed from 0 to 1, leader changed from <none> to f2d027e0ea9c4e96bbbdfb676999a27b (127.27.153.67). New cstate: current_term: 1 leader_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } health_report { overall_health: HEALTHY } } }
I20260810 03:50:26.267809 32349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.263870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:37723 (local address 127.0.0.1:60290)
0810 03:50:26.264129 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:26.264143 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:26.264270 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:26.264575 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:26.264578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:26.264586 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:50:26.264702 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.264709 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.265426 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.265429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.266101 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.266110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.266560 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.266563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.266668 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.266732 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:50:26.267414 (+ 682us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:50:26.267432 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 03:50:26.267549 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 03:50:26.267652 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260810 03:50:26.267818 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.263987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60290 (local address 127.27.153.67:37723)
0810 03:50:26.264121 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:26.264126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:26.264198 (+ 72us) server_negotiation.cc:407] Connection header received
0810 03:50:26.264407 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:26.264410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:26.264467 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:26.264552 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:50:26.264801 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.265315 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.266221 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.266452 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.266544 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.266881 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:50:26.267320 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:50:26.267426 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:26.267588 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 03:50:26.267676 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:50:26.273895 31976 raft_consensus.cc:1261] T e80fca4f6b73409dbd3bc2085bc3dee6 P 592678774ca74107a42f086d26a5563a [term 1 FOLLOWER]: Refusing update from remote peer f2d027e0ea9c4e96bbbdfb676999a27b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:26.273905 31732 raft_consensus.cc:1261] T e80fca4f6b73409dbd3bc2085bc3dee6 P 83c344f979094f4f9323e8a15deffa5e [term 1 FOLLOWER]: Refusing update from remote peer f2d027e0ea9c4e96bbbdfb676999a27b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:26.274714 32336 consensus_queue.cc:1035] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:26.275254 32336 consensus_queue.cc:1035] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b [LEADER]: Connected to new peer: Peer: permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:26.278863 32332 tablet_replica.cc:439] T e80fca4f6b73409dbd3bc2085bc3dee6 P 592678774ca74107a42f086d26a5563a: TxnStatusTablet state changed. Reason: New leader f2d027e0ea9c4e96bbbdfb676999a27b. Latest consensus state: current_term: 1 leader_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } } }
I20260810 03:50:26.278995 32332 tablet_replica.cc:442] T e80fca4f6b73409dbd3bc2085bc3dee6 P 592678774ca74107a42f086d26a5563a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:50:26.279666 32331 tablet_replica.cc:439] T e80fca4f6b73409dbd3bc2085bc3dee6 P 83c344f979094f4f9323e8a15deffa5e: TxnStatusTablet state changed. Reason: New leader f2d027e0ea9c4e96bbbdfb676999a27b. Latest consensus state: current_term: 1 leader_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } } }
I20260810 03:50:26.279798 32331 tablet_replica.cc:442] T e80fca4f6b73409dbd3bc2085bc3dee6 P 83c344f979094f4f9323e8a15deffa5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:50:26.283255 32336 tablet_replica.cc:439] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } } }
I20260810 03:50:26.283401 32336 tablet_replica.cc:442] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:50:26.284538 32332 tablet_replica.cc:439] T e80fca4f6b73409dbd3bc2085bc3dee6 P 592678774ca74107a42f086d26a5563a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } } }
I20260810 03:50:26.284672 32332 tablet_replica.cc:442] T e80fca4f6b73409dbd3bc2085bc3dee6 P 592678774ca74107a42f086d26a5563a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:50:26.285049 32368 mvcc.cc:205] Tried to move back new op lower bound from 7316823352409972736 to 7316823352327004160. Current Snapshot: MvccSnapshot[applied={T|T < 7316823352409972736}]
I20260810 03:50:26.293032 32333 tablet_replica.cc:439] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } } }
I20260810 03:50:26.293212 32333 tablet_replica.cc:442] T e80fca4f6b73409dbd3bc2085bc3dee6 P f2d027e0ea9c4e96bbbdfb676999a27b: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:50:26.293848 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.289887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.27.153.67:37723)
0810 03:50:26.290017 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:26.290021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:26.290036 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:50:26.290084 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:26.290087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:26.290136 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:26.290218 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:50:26.290491 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.291206 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.292265 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.292618 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.292726 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.293092 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:50:26.293181 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:50:26.293238 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:26.293639 (+ 401us) server_negotiation.cc:299] Negotiation successful
0810 03:50:26.293714 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 03:50:26.295284 32331 tablet_replica.cc:439] T e80fca4f6b73409dbd3bc2085bc3dee6 P 83c344f979094f4f9323e8a15deffa5e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } } }
I20260810 03:50:26.295445 32331 tablet_replica.cc:442] T e80fca4f6b73409dbd3bc2085bc3dee6 P 83c344f979094f4f9323e8a15deffa5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:50:26.296999 32367 mvcc.cc:205] Tried to move back new op lower bound from 7316823352409972736 to 7316823352327004160. Current Snapshot: MvccSnapshot[applied={T|T < 7316823352409972736}]
I20260810 03:50:26.306943 32294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.303263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:37723 (local address 127.27.153.67:49717)
0810 03:50:26.303429 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:26.303445 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:26.303570 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:26.303795 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:26.303797 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:26.303803 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:50:26.303906 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.303911 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.304861 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.304864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.305489 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.305498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.306084 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.306088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.306172 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.306239 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:50:26.306624 (+ 385us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:50:26.306630 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 03:50:26.306682 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 03:50:26.306762 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":1}
I20260810 03:50:26.307176 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.303338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:49717 (local address 127.27.153.67:37723)
0810 03:50:26.303547 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:26.303552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:26.303567 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:50:26.303645 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:26.303655 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:26.303707 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:26.303778 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:50:26.304000 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.304765 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.305592 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.305934 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.306057 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.306405 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:50:26.306548 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:50:26.306618 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:26.306695 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 03:50:26.306748 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 03:50:26.314333 31976 raft_consensus.cc:1261] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 592678774ca74107a42f086d26a5563a [term 1 FOLLOWER]: Refusing update from remote peer f2d027e0ea9c4e96bbbdfb676999a27b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:26.314349 31732 raft_consensus.cc:1261] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 83c344f979094f4f9323e8a15deffa5e [term 1 FOLLOWER]: Refusing update from remote peer f2d027e0ea9c4e96bbbdfb676999a27b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:26.314742 32333 consensus_queue.cc:1035] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b [LEADER]: Connected to new peer: Peer: permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:26.314936 32381 consensus_queue.cc:1035] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:26.318785 32366 mvcc.cc:205] Tried to move back new op lower bound from 7316823352579727360 to 7316823351660630016. Current Snapshot: MvccSnapshot[applied={T|T < 7316823352579727360}]
I20260810 03:50:26.357553 31444 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:26.357784 31444 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:26.360773 31444 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390: Sending DeleteTablet for 3 replicas of tablet 87ea02d8f6fc4a30ad9fd54f7c228914
I20260810 03:50:26.361603 32173 tablet_service.cc:1514] Processing DeleteTablet for tablet 87ea02d8f6fc4a30ad9fd54f7c228914 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:26 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:60278
I20260810 03:50:26.361807 31695 tablet_service.cc:1514] Processing DeleteTablet for tablet 87ea02d8f6fc4a30ad9fd54f7c228914 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:26 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:46752
I20260810 03:50:26.362246 32395 tablet_replica.cc:330] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b: stopping tablet replica
I20260810 03:50:26.362226 31942 tablet_service.cc:1514] Processing DeleteTablet for tablet 87ea02d8f6fc4a30ad9fd54f7c228914 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:26 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:59560
I20260810 03:50:26.362596 32395 raft_consensus.cc:2229] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:50:26.362757 32397 tablet_replica.cc:330] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 592678774ca74107a42f086d26a5563a: stopping tablet replica
I20260810 03:50:26.362972 32395 raft_consensus.cc:2258] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:26.363181 32397 raft_consensus.cc:2229] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 592678774ca74107a42f086d26a5563a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:26.363508 32397 raft_consensus.cc:2258] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 592678774ca74107a42f086d26a5563a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:26.363726 31444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:50:26.364215 31444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:50:26.364900 32395 ts_tablet_manager.cc:1905] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:26.365427 32397 ts_tablet_manager.cc:1905] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 592678774ca74107a42f086d26a5563a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:26.366106 32396 tablet_replica.cc:330] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 83c344f979094f4f9323e8a15deffa5e: stopping tablet replica
I20260810 03:50:26.366459 32396 raft_consensus.cc:2229] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 83c344f979094f4f9323e8a15deffa5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:26.366691 32396 raft_consensus.cc:2258] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 83c344f979094f4f9323e8a15deffa5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:26.368412 32395 ts_tablet_manager.cc:1918] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:50:26.368531 32395 log.cc:1196] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/87ea02d8f6fc4a30ad9fd54f7c228914
I20260810 03:50:26.368872 32395 ts_tablet_manager.cc:1939] T 87ea02d8f6fc4a30ad9fd54f7c228914 P f2d027e0ea9c4e96bbbdfb676999a27b: Deleting consensus metadata
I20260810 03:50:26.370445 31695 tablet_service.cc:1467] Processing CreateTablet for tablet 1ec296b7119a4dac9117aecacaf26b4a (DEFAULT_TABLE table=test-table [id=7605052f22df4ba0966a4bf3da7dba8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:26.370443 32173 tablet_service.cc:1467] Processing CreateTablet for tablet 1ec296b7119a4dac9117aecacaf26b4a (DEFAULT_TABLE table=test-table [id=7605052f22df4ba0966a4bf3da7dba8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:26.370733 31695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ec296b7119a4dac9117aecacaf26b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:26.370733 32173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ec296b7119a4dac9117aecacaf26b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:26.371511 32396 ts_tablet_manager.cc:1905] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 83c344f979094f4f9323e8a15deffa5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:26.373379 31942 tablet_service.cc:1467] Processing CreateTablet for tablet 1ec296b7119a4dac9117aecacaf26b4a (DEFAULT_TABLE table=test-table [id=7605052f22df4ba0966a4bf3da7dba8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:26.373449 31417 catalog_manager.cc:4841] TS f2d027e0ea9c4e96bbbdfb676999a27b (127.27.153.67:37723): tablet 87ea02d8f6fc4a30ad9fd54f7c228914 (table test-table [id=e4266eb33f5048f0b61532932b5b14e3]) successfully deleted
I20260810 03:50:26.373689 31942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ec296b7119a4dac9117aecacaf26b4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:26.374656 32396 ts_tablet_manager.cc:1918] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 83c344f979094f4f9323e8a15deffa5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:50:26.374801 32396 log.cc:1196] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 83c344f979094f4f9323e8a15deffa5e: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-0/wal/wals/87ea02d8f6fc4a30ad9fd54f7c228914
I20260810 03:50:26.375133 32396 ts_tablet_manager.cc:1939] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 83c344f979094f4f9323e8a15deffa5e: Deleting consensus metadata
I20260810 03:50:26.375629 32323 tablet_bootstrap.cc:492] T 1ec296b7119a4dac9117aecacaf26b4a P f2d027e0ea9c4e96bbbdfb676999a27b: Bootstrap starting.
I20260810 03:50:26.376703 32323 tablet_bootstrap.cc:654] T 1ec296b7119a4dac9117aecacaf26b4a P f2d027e0ea9c4e96bbbdfb676999a27b: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:26.377401 32322 tablet_bootstrap.cc:492] T 1ec296b7119a4dac9117aecacaf26b4a P 83c344f979094f4f9323e8a15deffa5e: Bootstrap starting.
I20260810 03:50:26.377452 32397 ts_tablet_manager.cc:1918] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 592678774ca74107a42f086d26a5563a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:50:26.377560 32397 log.cc:1196] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 592678774ca74107a42f086d26a5563a: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/87ea02d8f6fc4a30ad9fd54f7c228914
I20260810 03:50:26.377924 32397 ts_tablet_manager.cc:1939] T 87ea02d8f6fc4a30ad9fd54f7c228914 P 592678774ca74107a42f086d26a5563a: Deleting consensus metadata
I20260810 03:50:26.378391 32322 tablet_bootstrap.cc:654] T 1ec296b7119a4dac9117aecacaf26b4a P 83c344f979094f4f9323e8a15deffa5e: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:26.379244 31421 catalog_manager.cc:4841] TS 592678774ca74107a42f086d26a5563a (127.27.153.66:43007): tablet 87ea02d8f6fc4a30ad9fd54f7c228914 (table test-table [id=e4266eb33f5048f0b61532932b5b14e3]) successfully deleted
I20260810 03:50:26.379346 31420 catalog_manager.cc:4841] TS 83c344f979094f4f9323e8a15deffa5e (127.27.153.65:38971): tablet 87ea02d8f6fc4a30ad9fd54f7c228914 (table test-table [id=e4266eb33f5048f0b61532932b5b14e3]) successfully deleted
I20260810 03:50:26.379695 32322 tablet_bootstrap.cc:492] T 1ec296b7119a4dac9117aecacaf26b4a P 83c344f979094f4f9323e8a15deffa5e: No bootstrap required, opened a new log
I20260810 03:50:26.379779 32322 ts_tablet_manager.cc:1397] T 1ec296b7119a4dac9117aecacaf26b4a P 83c344f979094f4f9323e8a15deffa5e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 03:50:26.380259 32324 tablet_bootstrap.cc:492] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a: Bootstrap starting.
I20260810 03:50:26.380237 32322 raft_consensus.cc:348] T 1ec296b7119a4dac9117aecacaf26b4a P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:26.380358 32322 raft_consensus.cc:374] T 1ec296b7119a4dac9117aecacaf26b4a P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:26.380391 32322 raft_consensus.cc:729] T 1ec296b7119a4dac9117aecacaf26b4a P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c344f979094f4f9323e8a15deffa5e, State: Initialized, Role: FOLLOWER
I20260810 03:50:26.380522 32322 consensus_queue.cc:260] T 1ec296b7119a4dac9117aecacaf26b4a P 83c344f979094f4f9323e8a15deffa5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:26.381121 32322 ts_tablet_manager.cc:1428] T 1ec296b7119a4dac9117aecacaf26b4a P 83c344f979094f4f9323e8a15deffa5e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 03:50:26.381192 32324 tablet_bootstrap.cc:654] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:26.381692 32323 tablet_bootstrap.cc:492] T 1ec296b7119a4dac9117aecacaf26b4a P f2d027e0ea9c4e96bbbdfb676999a27b: No bootstrap required, opened a new log
I20260810 03:50:26.381801 32323 ts_tablet_manager.cc:1397] T 1ec296b7119a4dac9117aecacaf26b4a P f2d027e0ea9c4e96bbbdfb676999a27b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260810 03:50:26.382280 32323 raft_consensus.cc:348] T 1ec296b7119a4dac9117aecacaf26b4a P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:26.382416 32323 raft_consensus.cc:374] T 1ec296b7119a4dac9117aecacaf26b4a P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:26.382452 32323 raft_consensus.cc:729] T 1ec296b7119a4dac9117aecacaf26b4a P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2d027e0ea9c4e96bbbdfb676999a27b, State: Initialized, Role: FOLLOWER
I20260810 03:50:26.382551 32324 tablet_bootstrap.cc:492] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a: No bootstrap required, opened a new log
I20260810 03:50:26.382594 32323 consensus_queue.cc:260] T 1ec296b7119a4dac9117aecacaf26b4a P f2d027e0ea9c4e96bbbdfb676999a27b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:26.382649 32324 ts_tablet_manager.cc:1397] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 03:50:26.383085 32324 raft_consensus.cc:348] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:26.383199 32324 raft_consensus.cc:374] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:26.383229 32324 raft_consensus.cc:729] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 592678774ca74107a42f086d26a5563a, State: Initialized, Role: FOLLOWER
I20260810 03:50:26.383342 32324 consensus_queue.cc:260] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:26.383755 32324 ts_tablet_manager.cc:1428] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:50:26.385032 32323 ts_tablet_manager.cc:1428] T 1ec296b7119a4dac9117aecacaf26b4a P f2d027e0ea9c4e96bbbdfb676999a27b: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260810 03:50:26.400898 32332 raft_consensus.cc:482] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:50:26.401046 32332 raft_consensus.cc:504] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:26.402170 32332 leader_election.cc:288] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83c344f979094f4f9323e8a15deffa5e (127.27.153.65:38971), f2d027e0ea9c4e96bbbdfb676999a27b (127.27.153.67:37723)
I20260810 03:50:26.407219 32408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.402478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:38971 (local address 127.27.153.66:40233)
0810 03:50:26.402892 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:26.402907 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:26.403083 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:26.403395 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:26.403398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:26.403417 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:26.403578 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.403586 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.404344 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.404348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.405572 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.405582 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.406723 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.406726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.406875 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.406900 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 03:50:26.406967 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 03:50:26.407050 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":97,"threads_started":1}
I20260810 03:50:26.407202 32316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.402601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:40233 (local address 127.27.153.65:38971)
0810 03:50:26.402743 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:26.402748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:26.402989 (+ 241us) server_negotiation.cc:407] Connection header received
0810 03:50:26.403238 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:26.403240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:26.403293 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:26.403373 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:26.403706 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.404237 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.405717 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.406392 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.406491 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.406549 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:26.407004 (+ 455us) server_negotiation.cc:299] Negotiation successful
0810 03:50:26.407091 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:50:26.407389 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.402916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:45049 (local address 127.27.153.67:37723)
0810 03:50:26.403036 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:26.403040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:26.403110 (+ 70us) server_negotiation.cc:407] Connection header received
0810 03:50:26.403313 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:26.403316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:26.403365 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:26.403430 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:26.403810 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.404548 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.405908 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.406541 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.406650 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.406700 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:26.407173 (+ 473us) server_negotiation.cc:299] Negotiation successful
0810 03:50:26.407264 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 03:50:26.407970 32407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.402509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:37723 (local address 127.27.153.66:45049)
0810 03:50:26.403010 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:26.403025 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:26.403144 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:26.403560 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:26.403562 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:26.403577 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:26.403687 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.403693 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.404664 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.404667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.405784 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.405793 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.406892 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.406896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.407014 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.407039 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 03:50:26.407633 (+ 594us) client_negotiation.cc:240] Negotiation successful
0810 03:50:26.407717 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":134,"threads_started":1}
I20260810 03:50:26.408147 31731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ec296b7119a4dac9117aecacaf26b4a" candidate_uuid: "592678774ca74107a42f086d26a5563a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c344f979094f4f9323e8a15deffa5e" is_pre_election: true
I20260810 03:50:26.408317 31731 raft_consensus.cc:2454] T 1ec296b7119a4dac9117aecacaf26b4a P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 592678774ca74107a42f086d26a5563a in term 0.
I20260810 03:50:26.408712 32203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ec296b7119a4dac9117aecacaf26b4a" candidate_uuid: "592678774ca74107a42f086d26a5563a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" is_pre_election: true
I20260810 03:50:26.408739 31857 leader_election.cc:302] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 592678774ca74107a42f086d26a5563a, 83c344f979094f4f9323e8a15deffa5e; no voters:
I20260810 03:50:26.408972 32203 raft_consensus.cc:2454] T 1ec296b7119a4dac9117aecacaf26b4a P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 592678774ca74107a42f086d26a5563a in term 0.
I20260810 03:50:26.409045 32332 raft_consensus.cc:2781] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:50:26.409097 32332 raft_consensus.cc:482] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:50:26.409144 32332 raft_consensus.cc:3037] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:26.410256 32332 raft_consensus.cc:504] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:26.410611 32332 leader_election.cc:288] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a [CANDIDATE]: Term 1 election: Requested vote from peers 83c344f979094f4f9323e8a15deffa5e (127.27.153.65:38971), f2d027e0ea9c4e96bbbdfb676999a27b (127.27.153.67:37723)
I20260810 03:50:26.411028 32203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ec296b7119a4dac9117aecacaf26b4a" candidate_uuid: "592678774ca74107a42f086d26a5563a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b"
I20260810 03:50:26.411029 31731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ec296b7119a4dac9117aecacaf26b4a" candidate_uuid: "592678774ca74107a42f086d26a5563a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c344f979094f4f9323e8a15deffa5e"
I20260810 03:50:26.411145 32203 raft_consensus.cc:3037] T 1ec296b7119a4dac9117aecacaf26b4a P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:26.411171 31731 raft_consensus.cc:3037] T 1ec296b7119a4dac9117aecacaf26b4a P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:26.412168 32203 raft_consensus.cc:2454] T 1ec296b7119a4dac9117aecacaf26b4a P f2d027e0ea9c4e96bbbdfb676999a27b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 592678774ca74107a42f086d26a5563a in term 1.
I20260810 03:50:26.412364 31731 raft_consensus.cc:2454] T 1ec296b7119a4dac9117aecacaf26b4a P 83c344f979094f4f9323e8a15deffa5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 592678774ca74107a42f086d26a5563a in term 1.
I20260810 03:50:26.412483 31855 leader_election.cc:302] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 592678774ca74107a42f086d26a5563a, f2d027e0ea9c4e96bbbdfb676999a27b; no voters:
I20260810 03:50:26.412681 32332 raft_consensus.cc:2781] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:26.412917 32332 raft_consensus.cc:686] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a [term 1 LEADER]: Becoming Leader. State: Replica: 592678774ca74107a42f086d26a5563a, State: Running, Role: LEADER
I20260810 03:50:26.413173 32332 consensus_queue.cc:237] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:26.415935 31444 catalog_manager.cc:5495] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a reported cstate change: term changed from 0 to 1, leader changed from <none> to 592678774ca74107a42f086d26a5563a (127.27.153.66). New cstate: current_term: 1 leader_uuid: "592678774ca74107a42f086d26a5563a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } health_report { overall_health: UNKNOWN } } }
I20260810 03:50:26.440204 32321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.435803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59564 (local address 127.27.153.66:43007)
0810 03:50:26.435926 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:26.435929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:26.436050 (+ 121us) server_negotiation.cc:407] Connection header received
0810 03:50:26.436232 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:26.436234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:26.436302 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:26.436392 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:50:26.436649 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.437166 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.438365 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.438720 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.438812 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.439109 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:50:26.439794 (+ 685us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:50:26.439896 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:26.440008 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 03:50:26.440067 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 03:50:26.453734 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.449053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:45915 (local address 127.27.153.67:37723)
0810 03:50:26.449184 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:26.449189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:26.449497 (+ 308us) server_negotiation.cc:407] Connection header received
0810 03:50:26.449625 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:26.449628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:26.449685 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:26.449768 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:50:26.449979 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.450782 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.452358 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.452712 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.452878 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.453250 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:50:26.453346 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:50:26.453413 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:26.453518 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 03:50:26.453575 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 03:50:26.453784 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.448650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:37723 (local address 127.27.153.66:45915)
0810 03:50:26.449411 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:26.449425 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:26.449534 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:26.449774 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:26.449776 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:26.449783 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:50:26.449893 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.449897 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.450912 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.450914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.452241 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.452253 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.452916 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.452919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.453010 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.453064 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:50:26.453430 (+ 366us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:50:26.453437 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:50:26.453498 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 03:50:26.453591 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":116,"threads_started":1}
I20260810 03:50:26.461270 32203 raft_consensus.cc:1261] T 1ec296b7119a4dac9117aecacaf26b4a P f2d027e0ea9c4e96bbbdfb676999a27b [term 1 FOLLOWER]: Refusing update from remote peer 592678774ca74107a42f086d26a5563a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:26.461755 31731 raft_consensus.cc:1261] T 1ec296b7119a4dac9117aecacaf26b4a P 83c344f979094f4f9323e8a15deffa5e [term 1 FOLLOWER]: Refusing update from remote peer 592678774ca74107a42f086d26a5563a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:26.461869 32332 consensus_queue.cc:1035] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:26.462177 32409 consensus_queue.cc:1035] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:26.487484 32294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.483117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:43007 (local address 127.27.153.67:43407)
0810 03:50:26.483247 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:26.483272 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:26.483354 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:26.483872 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:26.483875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:26.483884 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:50:26.483985 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.483992 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.485013 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.485017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.486008 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:26.486019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.486705 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.486709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:26.486793 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.486828 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:50:26.487140 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:50:26.487146 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 03:50:26.487214 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 03:50:26.487308 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 03:50:26.488143 32321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:26.483434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:43407 (local address 127.27.153.66:43007)
0810 03:50:26.483569 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:26.483574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:26.483587 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:50:26.483632 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:26.483635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:26.483686 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:26.483757 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:50:26.484104 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.484896 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.486134 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:26.486520 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:26.486618 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:26.486957 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:50:26.487049 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:50:26.487105 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:26.487959 (+ 854us) server_negotiation.cc:299] Negotiation successful
0810 03:50:26.488020 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 03:50:26.488860 32368 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 03:50:28.055565 31444 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:28.055732 31444 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:28.058422 31444 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390: Sending DeleteTablet for 3 replicas of tablet 1ec296b7119a4dac9117aecacaf26b4a
I20260810 03:50:28.059098 31695 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ec296b7119a4dac9117aecacaf26b4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:28 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:46752
I20260810 03:50:28.059124 31942 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ec296b7119a4dac9117aecacaf26b4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:28 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:59560
I20260810 03:50:28.059239 32173 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ec296b7119a4dac9117aecacaf26b4a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:28 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:60278
I20260810 03:50:28.059531 32470 tablet_replica.cc:330] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a: stopping tablet replica
I20260810 03:50:28.059599 32474 tablet_replica.cc:330] T 1ec296b7119a4dac9117aecacaf26b4a P f2d027e0ea9c4e96bbbdfb676999a27b: stopping tablet replica
I20260810 03:50:28.059788 32473 tablet_replica.cc:330] T 1ec296b7119a4dac9117aecacaf26b4a P 83c344f979094f4f9323e8a15deffa5e: stopping tablet replica
I20260810 03:50:28.059954 32474 raft_consensus.cc:2229] T 1ec296b7119a4dac9117aecacaf26b4a P f2d027e0ea9c4e96bbbdfb676999a27b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:28.060094 32474 raft_consensus.cc:2258] T 1ec296b7119a4dac9117aecacaf26b4a P f2d027e0ea9c4e96bbbdfb676999a27b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:28.060173 32470 raft_consensus.cc:2229] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:50:28.060307 32473 raft_consensus.cc:2229] T 1ec296b7119a4dac9117aecacaf26b4a P 83c344f979094f4f9323e8a15deffa5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:28.060472 32473 raft_consensus.cc:2258] T 1ec296b7119a4dac9117aecacaf26b4a P 83c344f979094f4f9323e8a15deffa5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:28.060565 32470 raft_consensus.cc:2258] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:28.061048 32474 ts_tablet_manager.cc:1905] T 1ec296b7119a4dac9117aecacaf26b4a P f2d027e0ea9c4e96bbbdfb676999a27b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:28.061077 32470 ts_tablet_manager.cc:1905] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:28.063862 32474 ts_tablet_manager.cc:1918] T 1ec296b7119a4dac9117aecacaf26b4a P f2d027e0ea9c4e96bbbdfb676999a27b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 03:50:28.063862 32470 ts_tablet_manager.cc:1918] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 03:50:28.063952 32474 log.cc:1196] T 1ec296b7119a4dac9117aecacaf26b4a P f2d027e0ea9c4e96bbbdfb676999a27b: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/1ec296b7119a4dac9117aecacaf26b4a
I20260810 03:50:28.063952 32470 log.cc:1196] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/1ec296b7119a4dac9117aecacaf26b4a
I20260810 03:50:28.063961 32473 ts_tablet_manager.cc:1905] T 1ec296b7119a4dac9117aecacaf26b4a P 83c344f979094f4f9323e8a15deffa5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:28.064208 32474 ts_tablet_manager.cc:1939] T 1ec296b7119a4dac9117aecacaf26b4a P f2d027e0ea9c4e96bbbdfb676999a27b: Deleting consensus metadata
I20260810 03:50:28.064219 32470 ts_tablet_manager.cc:1939] T 1ec296b7119a4dac9117aecacaf26b4a P 592678774ca74107a42f086d26a5563a: Deleting consensus metadata
I20260810 03:50:28.065304 31421 catalog_manager.cc:4841] TS 592678774ca74107a42f086d26a5563a (127.27.153.66:43007): tablet 1ec296b7119a4dac9117aecacaf26b4a (table test-table [id=7605052f22df4ba0966a4bf3da7dba8f]) successfully deleted
I20260810 03:50:28.065304 31417 catalog_manager.cc:4841] TS f2d027e0ea9c4e96bbbdfb676999a27b (127.27.153.67:37723): tablet 1ec296b7119a4dac9117aecacaf26b4a (table test-table [id=7605052f22df4ba0966a4bf3da7dba8f]) successfully deleted
I20260810 03:50:28.066462 32473 ts_tablet_manager.cc:1918] T 1ec296b7119a4dac9117aecacaf26b4a P 83c344f979094f4f9323e8a15deffa5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 03:50:28.066588 32473 log.cc:1196] T 1ec296b7119a4dac9117aecacaf26b4a P 83c344f979094f4f9323e8a15deffa5e: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-0/wal/wals/1ec296b7119a4dac9117aecacaf26b4a
I20260810 03:50:28.066838 32473 ts_tablet_manager.cc:1939] T 1ec296b7119a4dac9117aecacaf26b4a P 83c344f979094f4f9323e8a15deffa5e: Deleting consensus metadata
I20260810 03:50:28.067533 31420 catalog_manager.cc:4841] TS 83c344f979094f4f9323e8a15deffa5e (127.27.153.65:38971): tablet 1ec296b7119a4dac9117aecacaf26b4a (table test-table [id=7605052f22df4ba0966a4bf3da7dba8f]) successfully deleted
Aug 10 03:50:28 dist-test-slave-pww0 krb5kdc[31368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333828, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 03:50:28 dist-test-slave-pww0 krb5kdc[31368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333828, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:28.089860 32489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:28.080601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.27.153.126:41135)
0810 03:50:28.080934 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:28.080943 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:28.080964 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:50:28.081134 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:28.081137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:28.081190 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:28.081274 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:28.081960 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:28.082538 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:28.083433 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:28.083776 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:28.083874 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:28.086343 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:28.086346 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:28.086349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:28.086385 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:28.088055 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:28.088526 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:28.088529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:28.088531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:28.088647 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:28.088911 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:28.088914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:28.088915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:28.089124 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:28.089234 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:28.089562 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 03:50:28.089719 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":112,"threads_started":1}
I20260810 03:50:28.092994 31444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:50:28.093502 31444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:50:28.098644 31695 tablet_service.cc:1467] Processing CreateTablet for tablet ef3c5330384e40d7b3433fead65606da (DEFAULT_TABLE table=test-table [id=de11c8cf4dfc46fa9a1b2742f1de5ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:28.098785 31942 tablet_service.cc:1467] Processing CreateTablet for tablet ef3c5330384e40d7b3433fead65606da (DEFAULT_TABLE table=test-table [id=de11c8cf4dfc46fa9a1b2742f1de5ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:28.098824 32173 tablet_service.cc:1467] Processing CreateTablet for tablet ef3c5330384e40d7b3433fead65606da (DEFAULT_TABLE table=test-table [id=de11c8cf4dfc46fa9a1b2742f1de5ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:28.099004 31695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef3c5330384e40d7b3433fead65606da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:28.099082 32173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef3c5330384e40d7b3433fead65606da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:28.099092 31942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef3c5330384e40d7b3433fead65606da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:28.101223 32490 tablet_bootstrap.cc:492] T ef3c5330384e40d7b3433fead65606da P 592678774ca74107a42f086d26a5563a: Bootstrap starting.
I20260810 03:50:28.101699 32492 tablet_bootstrap.cc:492] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e: Bootstrap starting.
I20260810 03:50:28.101977 32493 tablet_bootstrap.cc:492] T ef3c5330384e40d7b3433fead65606da P f2d027e0ea9c4e96bbbdfb676999a27b: Bootstrap starting.
I20260810 03:50:28.102663 32492 tablet_bootstrap.cc:654] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:28.102727 32490 tablet_bootstrap.cc:654] T ef3c5330384e40d7b3433fead65606da P 592678774ca74107a42f086d26a5563a: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:28.102861 32493 tablet_bootstrap.cc:654] T ef3c5330384e40d7b3433fead65606da P f2d027e0ea9c4e96bbbdfb676999a27b: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:28.103945 32493 tablet_bootstrap.cc:492] T ef3c5330384e40d7b3433fead65606da P f2d027e0ea9c4e96bbbdfb676999a27b: No bootstrap required, opened a new log
I20260810 03:50:28.103946 32490 tablet_bootstrap.cc:492] T ef3c5330384e40d7b3433fead65606da P 592678774ca74107a42f086d26a5563a: No bootstrap required, opened a new log
I20260810 03:50:28.104024 32493 ts_tablet_manager.cc:1397] T ef3c5330384e40d7b3433fead65606da P f2d027e0ea9c4e96bbbdfb676999a27b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 03:50:28.104032 32490 ts_tablet_manager.cc:1397] T ef3c5330384e40d7b3433fead65606da P 592678774ca74107a42f086d26a5563a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 03:50:28.104429 32493 raft_consensus.cc:348] T ef3c5330384e40d7b3433fead65606da P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:28.104470 32490 raft_consensus.cc:348] T ef3c5330384e40d7b3433fead65606da P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:28.104552 32493 raft_consensus.cc:374] T ef3c5330384e40d7b3433fead65606da P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:28.104570 32490 raft_consensus.cc:374] T ef3c5330384e40d7b3433fead65606da P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:28.104586 32493 raft_consensus.cc:729] T ef3c5330384e40d7b3433fead65606da P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2d027e0ea9c4e96bbbdfb676999a27b, State: Initialized, Role: FOLLOWER
I20260810 03:50:28.104600 32490 raft_consensus.cc:729] T ef3c5330384e40d7b3433fead65606da P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 592678774ca74107a42f086d26a5563a, State: Initialized, Role: FOLLOWER
I20260810 03:50:28.104707 32493 consensus_queue.cc:260] T ef3c5330384e40d7b3433fead65606da P f2d027e0ea9c4e96bbbdfb676999a27b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:28.104720 32490 consensus_queue.cc:260] T ef3c5330384e40d7b3433fead65606da P 592678774ca74107a42f086d26a5563a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:28.104982 32490 ts_tablet_manager.cc:1428] T ef3c5330384e40d7b3433fead65606da P 592678774ca74107a42f086d26a5563a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:50:28.105391 32493 ts_tablet_manager.cc:1428] T ef3c5330384e40d7b3433fead65606da P f2d027e0ea9c4e96bbbdfb676999a27b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:50:28.105574 32492 tablet_bootstrap.cc:492] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e: No bootstrap required, opened a new log
I20260810 03:50:28.105669 32492 ts_tablet_manager.cc:1397] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 03:50:28.106103 32492 raft_consensus.cc:348] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:28.106215 32492 raft_consensus.cc:374] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:28.106244 32492 raft_consensus.cc:729] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c344f979094f4f9323e8a15deffa5e, State: Initialized, Role: FOLLOWER
I20260810 03:50:28.106374 32492 consensus_queue.cc:260] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:28.106789 32492 ts_tablet_manager.cc:1428] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 03:50:28.376463 32501 raft_consensus.cc:482] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:50:28.376667 32501 raft_consensus.cc:504] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:28.377826 32501 leader_election.cc:288] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 592678774ca74107a42f086d26a5563a (127.27.153.66:43007), f2d027e0ea9c4e96bbbdfb676999a27b (127.27.153.67:37723)
I20260810 03:50:28.382157 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:28.378251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:54045 (local address 127.27.153.67:37723)
0810 03:50:28.378543 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:28.378552 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:28.378717 (+ 165us) server_negotiation.cc:407] Connection header received
0810 03:50:28.378836 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:28.378838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:28.378882 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:28.378947 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:28.379294 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:28.379920 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:28.381069 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:28.381483 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:28.381588 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:28.381647 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:28.381906 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 03:50:28.381991 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":102,"threads_started":1}
I20260810 03:50:28.382158 32514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:28.378205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:37199 (local address 127.27.153.66:43007)
0810 03:50:28.378561 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:28.378573 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:28.378675 (+ 102us) server_negotiation.cc:407] Connection header received
0810 03:50:28.378836 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:28.378840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:28.378891 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:28.378981 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:28.379289 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:28.379917 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:28.381089 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:28.381503 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:28.381629 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:28.381693 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:28.381903 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 03:50:28.381991 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":137,"threads_started":1}
I20260810 03:50:28.382175 32504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:28.378078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:43007 (local address 127.27.153.65:37199)
0810 03:50:28.378539 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:28.378554 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:28.378712 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:28.378998 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:28.379001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:28.379018 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:28.379167 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:28.379173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:28.380034 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:28.380037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:28.380952 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:28.380962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:28.381630 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:28.381634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:28.381783 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:28.381804 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 03:50:28.381869 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 03:50:28.381976 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":103,"spinlock_wait_cycles":896,"thread_start_us":152,"threads_started":1}
I20260810 03:50:28.382186 32512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:28.378123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:37723 (local address 127.27.153.65:54045)
0810 03:50:28.378588 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:28.378604 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:28.378728 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:28.378989 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:28.378992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:28.379010 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:28.379172 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:28.379178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:28.380029 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:28.380033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:28.380933 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:28.380944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:28.381615 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:28.381619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:28.381774 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:28.381800 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 03:50:28.381878 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 03:50:28.381971 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":33,"spinlock_wait_cycles":6528,"thread_start_us":140,"threads_started":1}
I20260810 03:50:28.382694 31976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3c5330384e40d7b3433fead65606da" candidate_uuid: "83c344f979094f4f9323e8a15deffa5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "592678774ca74107a42f086d26a5563a" is_pre_election: true
I20260810 03:50:28.382869 31976 raft_consensus.cc:2454] T ef3c5330384e40d7b3433fead65606da P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c344f979094f4f9323e8a15deffa5e in term 0.
I20260810 03:50:28.383033 32203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3c5330384e40d7b3433fead65606da" candidate_uuid: "83c344f979094f4f9323e8a15deffa5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" is_pre_election: true
I20260810 03:50:28.383168 32203 raft_consensus.cc:2454] T ef3c5330384e40d7b3433fead65606da P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c344f979094f4f9323e8a15deffa5e in term 0.
I20260810 03:50:28.383289 31616 leader_election.cc:302] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 592678774ca74107a42f086d26a5563a, 83c344f979094f4f9323e8a15deffa5e; no voters:
I20260810 03:50:28.383571 32501 raft_consensus.cc:2781] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:50:28.383636 32501 raft_consensus.cc:482] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:50:28.383679 32501 raft_consensus.cc:3037] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:28.384753 32501 raft_consensus.cc:504] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:28.385111 32501 leader_election.cc:288] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e [CANDIDATE]: Term 1 election: Requested vote from peers 592678774ca74107a42f086d26a5563a (127.27.153.66:43007), f2d027e0ea9c4e96bbbdfb676999a27b (127.27.153.67:37723)
I20260810 03:50:28.385509 32203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3c5330384e40d7b3433fead65606da" candidate_uuid: "83c344f979094f4f9323e8a15deffa5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b"
I20260810 03:50:28.385488 31976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef3c5330384e40d7b3433fead65606da" candidate_uuid: "83c344f979094f4f9323e8a15deffa5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "592678774ca74107a42f086d26a5563a"
I20260810 03:50:28.385630 32203 raft_consensus.cc:3037] T ef3c5330384e40d7b3433fead65606da P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:28.385630 31976 raft_consensus.cc:3037] T ef3c5330384e40d7b3433fead65606da P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:28.386816 31976 raft_consensus.cc:2454] T ef3c5330384e40d7b3433fead65606da P 592678774ca74107a42f086d26a5563a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c344f979094f4f9323e8a15deffa5e in term 1.
I20260810 03:50:28.386816 32203 raft_consensus.cc:2454] T ef3c5330384e40d7b3433fead65606da P f2d027e0ea9c4e96bbbdfb676999a27b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c344f979094f4f9323e8a15deffa5e in term 1.
I20260810 03:50:28.387095 31616 leader_election.cc:302] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 592678774ca74107a42f086d26a5563a, 83c344f979094f4f9323e8a15deffa5e; no voters:
I20260810 03:50:28.387305 32501 raft_consensus.cc:2781] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:28.387553 32501 raft_consensus.cc:686] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e [term 1 LEADER]: Becoming Leader. State: Replica: 83c344f979094f4f9323e8a15deffa5e, State: Running, Role: LEADER
I20260810 03:50:28.387830 32501 consensus_queue.cc:237] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:28.390172 31444 catalog_manager.cc:5495] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 83c344f979094f4f9323e8a15deffa5e (127.27.153.65). New cstate: current_term: 1 leader_uuid: "83c344f979094f4f9323e8a15deffa5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } health_report { overall_health: UNKNOWN } } }
I20260810 03:50:28.412214 32519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:28.408558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46768 (local address 127.27.153.65:38971)
0810 03:50:28.408838 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:28.408845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:28.408858 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:50:28.408934 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:28.408936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:28.408989 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:28.409093 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:50:28.409327 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:28.409875 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:28.410682 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:28.410929 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:28.411002 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:28.411320 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:50:28.411858 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:50:28.411949 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:28.412032 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 03:50:28.412082 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":110,"threads_started":1}
I20260810 03:50:28.423120 32515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:28.419949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:56363 (local address 127.27.153.67:37723)
0810 03:50:28.420070 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:28.420074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:28.420087 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:50:28.420202 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:28.420205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:28.420247 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:28.420308 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:50:28.420593 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:28.421125 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:28.421975 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:28.422209 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:28.422327 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:28.422640 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:50:28.422715 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:50:28.422819 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:28.422915 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 03:50:28.422973 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 03:50:28.423133 32522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:28.419705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:37723 (local address 127.27.153.65:56363)
0810 03:50:28.420018 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:28.420030 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:28.420118 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:28.420323 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:28.420325 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:28.420331 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:50:28.420438 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:28.420443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:28.421236 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:28.421239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:28.421860 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:28.421869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:28.422369 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:28.422374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:28.422461 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:28.422511 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:50:28.422830 (+ 319us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:50:28.422837 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:50:28.422897 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 03:50:28.422978 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":108,"threads_started":1}
I20260810 03:50:28.429291 32203 raft_consensus.cc:1261] T ef3c5330384e40d7b3433fead65606da P f2d027e0ea9c4e96bbbdfb676999a27b [term 1 FOLLOWER]: Refusing update from remote peer 83c344f979094f4f9323e8a15deffa5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:28.429342 31976 raft_consensus.cc:1261] T ef3c5330384e40d7b3433fead65606da P 592678774ca74107a42f086d26a5563a [term 1 FOLLOWER]: Refusing update from remote peer 83c344f979094f4f9323e8a15deffa5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:28.429775 32501 consensus_queue.cc:1035] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:28.429979 32516 consensus_queue.cc:1035] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:28.448736 32534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:28.444983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:38971 (local address 127.27.153.67:45683)
0810 03:50:28.445294 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:28.445306 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:28.445407 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:28.445616 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:28.445619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:28.445627 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:50:28.445735 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:28.445741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:28.446664 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:28.446667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:28.447511 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:28.447523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:28.448040 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:28.448043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:28.448124 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:28.448156 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:50:28.448441 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:50:28.448454 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 03:50:28.448523 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 03:50:28.448603 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":121,"threads_started":1}
I20260810 03:50:28.448774 32519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:28.445061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:45683 (local address 127.27.153.65:38971)
0810 03:50:28.445169 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:28.445174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:28.445381 (+ 207us) server_negotiation.cc:407] Connection header received
0810 03:50:28.445493 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:28.445496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:28.445540 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:28.445599 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:50:28.445835 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:28.446417 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:28.447612 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:28.447921 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:28.448025 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:28.448257 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:50:28.448324 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:50:28.448374 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:28.448605 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 03:50:28.448664 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 03:50:28.473273 31444 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:28.473441 31444 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:28.475734 31444 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390: Sending DeleteTablet for 3 replicas of tablet ef3c5330384e40d7b3433fead65606da
I20260810 03:50:28.476548 31942 tablet_service.cc:1514] Processing DeleteTablet for tablet ef3c5330384e40d7b3433fead65606da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:28 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:59560
I20260810 03:50:28.476572 32173 tablet_service.cc:1514] Processing DeleteTablet for tablet ef3c5330384e40d7b3433fead65606da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:28 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:60278
I20260810 03:50:28.476764 32470 tablet_replica.cc:330] T ef3c5330384e40d7b3433fead65606da P 592678774ca74107a42f086d26a5563a: stopping tablet replica
I20260810 03:50:28.476814 32474 tablet_replica.cc:330] T ef3c5330384e40d7b3433fead65606da P f2d027e0ea9c4e96bbbdfb676999a27b: stopping tablet replica
I20260810 03:50:28.476861 32470 raft_consensus.cc:2229] T ef3c5330384e40d7b3433fead65606da P 592678774ca74107a42f086d26a5563a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:28.476961 32470 raft_consensus.cc:2258] T ef3c5330384e40d7b3433fead65606da P 592678774ca74107a42f086d26a5563a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:28.476964 32474 raft_consensus.cc:2229] T ef3c5330384e40d7b3433fead65606da P f2d027e0ea9c4e96bbbdfb676999a27b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:28.477090 32474 raft_consensus.cc:2258] T ef3c5330384e40d7b3433fead65606da P f2d027e0ea9c4e96bbbdfb676999a27b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:28.477375 31695 tablet_service.cc:1514] Processing DeleteTablet for tablet ef3c5330384e40d7b3433fead65606da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:28 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:46752
I20260810 03:50:28.477550 32473 tablet_replica.cc:330] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e: stopping tablet replica
I20260810 03:50:28.477644 32473 raft_consensus.cc:2229] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:50:28.477771 32470 ts_tablet_manager.cc:1905] T ef3c5330384e40d7b3433fead65606da P 592678774ca74107a42f086d26a5563a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:28.477892 32474 ts_tablet_manager.cc:1905] T ef3c5330384e40d7b3433fead65606da P f2d027e0ea9c4e96bbbdfb676999a27b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:28.477967 32473 raft_consensus.cc:2258] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:28.478785 32473 ts_tablet_manager.cc:1905] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:28.480647 32474 ts_tablet_manager.cc:1918] T ef3c5330384e40d7b3433fead65606da P f2d027e0ea9c4e96bbbdfb676999a27b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:50:28.480732 32474 log.cc:1196] T ef3c5330384e40d7b3433fead65606da P f2d027e0ea9c4e96bbbdfb676999a27b: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/ef3c5330384e40d7b3433fead65606da
I20260810 03:50:28.480713 32470 ts_tablet_manager.cc:1918] T ef3c5330384e40d7b3433fead65606da P 592678774ca74107a42f086d26a5563a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:50:28.480793 32470 log.cc:1196] T ef3c5330384e40d7b3433fead65606da P 592678774ca74107a42f086d26a5563a: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/ef3c5330384e40d7b3433fead65606da
I20260810 03:50:28.481007 32474 ts_tablet_manager.cc:1939] T ef3c5330384e40d7b3433fead65606da P f2d027e0ea9c4e96bbbdfb676999a27b: Deleting consensus metadata
I20260810 03:50:28.481060 32470 ts_tablet_manager.cc:1939] T ef3c5330384e40d7b3433fead65606da P 592678774ca74107a42f086d26a5563a: Deleting consensus metadata
I20260810 03:50:28.481582 32473 ts_tablet_manager.cc:1918] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:50:28.481653 32473 log.cc:1196] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-0/wal/wals/ef3c5330384e40d7b3433fead65606da
I20260810 03:50:28.481806 31417 catalog_manager.cc:4841] TS f2d027e0ea9c4e96bbbdfb676999a27b (127.27.153.67:37723): tablet ef3c5330384e40d7b3433fead65606da (table test-table [id=de11c8cf4dfc46fa9a1b2742f1de5ed9]) successfully deleted
I20260810 03:50:28.481884 32473 ts_tablet_manager.cc:1939] T ef3c5330384e40d7b3433fead65606da P 83c344f979094f4f9323e8a15deffa5e: Deleting consensus metadata
I20260810 03:50:28.481801 31421 catalog_manager.cc:4841] TS 592678774ca74107a42f086d26a5563a (127.27.153.66:43007): tablet ef3c5330384e40d7b3433fead65606da (table test-table [id=de11c8cf4dfc46fa9a1b2742f1de5ed9]) successfully deleted
I20260810 03:50:28.483093 31444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 03:50:28.483219 31420 catalog_manager.cc:4841] TS 83c344f979094f4f9323e8a15deffa5e (127.27.153.65:38971): tablet ef3c5330384e40d7b3433fead65606da (table test-table [id=de11c8cf4dfc46fa9a1b2742f1de5ed9]) successfully deleted
W20260810 03:50:28.483682 31444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:50:28.489444 32173 tablet_service.cc:1467] Processing CreateTablet for tablet b6a00c6e864e4242b93b214d372eea59 (DEFAULT_TABLE table=test-table [id=37c24790761949168f7e9c1ac333e7e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:28.489444 31942 tablet_service.cc:1467] Processing CreateTablet for tablet b6a00c6e864e4242b93b214d372eea59 (DEFAULT_TABLE table=test-table [id=37c24790761949168f7e9c1ac333e7e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:28.489781 31942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6a00c6e864e4242b93b214d372eea59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:28.489781 32173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6a00c6e864e4242b93b214d372eea59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:28.492410 32493 tablet_bootstrap.cc:492] T b6a00c6e864e4242b93b214d372eea59 P f2d027e0ea9c4e96bbbdfb676999a27b: Bootstrap starting.
I20260810 03:50:28.493145 32493 tablet_bootstrap.cc:654] T b6a00c6e864e4242b93b214d372eea59 P f2d027e0ea9c4e96bbbdfb676999a27b: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:28.494062 32490 tablet_bootstrap.cc:492] T b6a00c6e864e4242b93b214d372eea59 P 592678774ca74107a42f086d26a5563a: Bootstrap starting.
I20260810 03:50:28.494751 31695 tablet_service.cc:1467] Processing CreateTablet for tablet b6a00c6e864e4242b93b214d372eea59 (DEFAULT_TABLE table=test-table [id=37c24790761949168f7e9c1ac333e7e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:28.495030 32490 tablet_bootstrap.cc:654] T b6a00c6e864e4242b93b214d372eea59 P 592678774ca74107a42f086d26a5563a: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:28.495033 31695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6a00c6e864e4242b93b214d372eea59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:28.496308 32490 tablet_bootstrap.cc:492] T b6a00c6e864e4242b93b214d372eea59 P 592678774ca74107a42f086d26a5563a: No bootstrap required, opened a new log
I20260810 03:50:28.496397 32490 ts_tablet_manager.cc:1397] T b6a00c6e864e4242b93b214d372eea59 P 592678774ca74107a42f086d26a5563a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260810 03:50:28.496807 32490 raft_consensus.cc:348] T b6a00c6e864e4242b93b214d372eea59 P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:28.496919 32490 raft_consensus.cc:374] T b6a00c6e864e4242b93b214d372eea59 P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:28.496949 32490 raft_consensus.cc:729] T b6a00c6e864e4242b93b214d372eea59 P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 592678774ca74107a42f086d26a5563a, State: Initialized, Role: FOLLOWER
I20260810 03:50:28.497092 32490 consensus_queue.cc:260] T b6a00c6e864e4242b93b214d372eea59 P 592678774ca74107a42f086d26a5563a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:28.497388 32490 ts_tablet_manager.cc:1428] T b6a00c6e864e4242b93b214d372eea59 P 592678774ca74107a42f086d26a5563a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 03:50:28.498183 32493 tablet_bootstrap.cc:492] T b6a00c6e864e4242b93b214d372eea59 P f2d027e0ea9c4e96bbbdfb676999a27b: No bootstrap required, opened a new log
I20260810 03:50:28.498294 32493 ts_tablet_manager.cc:1397] T b6a00c6e864e4242b93b214d372eea59 P f2d027e0ea9c4e96bbbdfb676999a27b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260810 03:50:28.498745 32493 raft_consensus.cc:348] T b6a00c6e864e4242b93b214d372eea59 P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:28.498869 32493 raft_consensus.cc:374] T b6a00c6e864e4242b93b214d372eea59 P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:28.498910 32493 raft_consensus.cc:729] T b6a00c6e864e4242b93b214d372eea59 P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2d027e0ea9c4e96bbbdfb676999a27b, State: Initialized, Role: FOLLOWER
I20260810 03:50:28.499051 32493 consensus_queue.cc:260] T b6a00c6e864e4242b93b214d372eea59 P f2d027e0ea9c4e96bbbdfb676999a27b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:28.499336 32493 ts_tablet_manager.cc:1428] T b6a00c6e864e4242b93b214d372eea59 P f2d027e0ea9c4e96bbbdfb676999a27b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 03:50:28.499775 32492 tablet_bootstrap.cc:492] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e: Bootstrap starting.
I20260810 03:50:28.500799 32492 tablet_bootstrap.cc:654] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:28.502741 32492 tablet_bootstrap.cc:492] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e: No bootstrap required, opened a new log
I20260810 03:50:28.502858 32492 ts_tablet_manager.cc:1397] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:50:28.503342 32492 raft_consensus.cc:348] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:28.503536 32492 raft_consensus.cc:374] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:28.503573 32492 raft_consensus.cc:729] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c344f979094f4f9323e8a15deffa5e, State: Initialized, Role: FOLLOWER
I20260810 03:50:28.503711 32492 consensus_queue.cc:260] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:28.504211 32492 ts_tablet_manager.cc:1428] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260810 03:50:28.562258 32526 raft_consensus.cc:482] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:50:28.562422 32526 raft_consensus.cc:504] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:28.562933 32526 leader_election.cc:288] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 592678774ca74107a42f086d26a5563a (127.27.153.66:43007), f2d027e0ea9c4e96bbbdfb676999a27b (127.27.153.67:37723)
I20260810 03:50:28.563494 31976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a00c6e864e4242b93b214d372eea59" candidate_uuid: "83c344f979094f4f9323e8a15deffa5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "592678774ca74107a42f086d26a5563a" is_pre_election: true
I20260810 03:50:28.563488 32203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a00c6e864e4242b93b214d372eea59" candidate_uuid: "83c344f979094f4f9323e8a15deffa5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" is_pre_election: true
I20260810 03:50:28.563663 31976 raft_consensus.cc:2454] T b6a00c6e864e4242b93b214d372eea59 P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c344f979094f4f9323e8a15deffa5e in term 0.
I20260810 03:50:28.563670 32203 raft_consensus.cc:2454] T b6a00c6e864e4242b93b214d372eea59 P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c344f979094f4f9323e8a15deffa5e in term 0.
I20260810 03:50:28.564015 31616 leader_election.cc:302] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 592678774ca74107a42f086d26a5563a, 83c344f979094f4f9323e8a15deffa5e; no voters:
I20260810 03:50:28.564224 32526 raft_consensus.cc:2781] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:50:28.564288 32526 raft_consensus.cc:482] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:50:28.564327 32526 raft_consensus.cc:3037] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:28.565461 32526 raft_consensus.cc:504] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:28.565965 32526 leader_election.cc:288] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e [CANDIDATE]: Term 1 election: Requested vote from peers 592678774ca74107a42f086d26a5563a (127.27.153.66:43007), f2d027e0ea9c4e96bbbdfb676999a27b (127.27.153.67:37723)
I20260810 03:50:28.566249 31976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a00c6e864e4242b93b214d372eea59" candidate_uuid: "83c344f979094f4f9323e8a15deffa5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "592678774ca74107a42f086d26a5563a"
I20260810 03:50:28.566359 31976 raft_consensus.cc:3037] T b6a00c6e864e4242b93b214d372eea59 P 592678774ca74107a42f086d26a5563a [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:28.566308 32203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6a00c6e864e4242b93b214d372eea59" candidate_uuid: "83c344f979094f4f9323e8a15deffa5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b"
I20260810 03:50:28.566442 32203 raft_consensus.cc:3037] T b6a00c6e864e4242b93b214d372eea59 P f2d027e0ea9c4e96bbbdfb676999a27b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:28.567391 31976 raft_consensus.cc:2454] T b6a00c6e864e4242b93b214d372eea59 P 592678774ca74107a42f086d26a5563a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c344f979094f4f9323e8a15deffa5e in term 1.
I20260810 03:50:28.567744 32203 raft_consensus.cc:2454] T b6a00c6e864e4242b93b214d372eea59 P f2d027e0ea9c4e96bbbdfb676999a27b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c344f979094f4f9323e8a15deffa5e in term 1.
I20260810 03:50:28.567843 31616 leader_election.cc:302] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 592678774ca74107a42f086d26a5563a, 83c344f979094f4f9323e8a15deffa5e; no voters:
I20260810 03:50:28.568109 32526 raft_consensus.cc:2781] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:28.568262 32526 raft_consensus.cc:686] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e [term 1 LEADER]: Becoming Leader. State: Replica: 83c344f979094f4f9323e8a15deffa5e, State: Running, Role: LEADER
I20260810 03:50:28.568457 32526 consensus_queue.cc:237] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } }
I20260810 03:50:28.570250 31444 catalog_manager.cc:5495] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e reported cstate change: term changed from 0 to 1, leader changed from <none> to 83c344f979094f4f9323e8a15deffa5e (127.27.153.65). New cstate: current_term: 1 leader_uuid: "83c344f979094f4f9323e8a15deffa5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c344f979094f4f9323e8a15deffa5e" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 } health_report { overall_health: UNKNOWN } } }
I20260810 03:50:28.600993 32203 raft_consensus.cc:1261] T b6a00c6e864e4242b93b214d372eea59 P f2d027e0ea9c4e96bbbdfb676999a27b [term 1 FOLLOWER]: Refusing update from remote peer 83c344f979094f4f9323e8a15deffa5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:28.601024 31976 raft_consensus.cc:1261] T b6a00c6e864e4242b93b214d372eea59 P 592678774ca74107a42f086d26a5563a [term 1 FOLLOWER]: Refusing update from remote peer 83c344f979094f4f9323e8a15deffa5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:28.601426 32527 consensus_queue.cc:1035] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "592678774ca74107a42f086d26a5563a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:28.601636 32501 consensus_queue.cc:1035] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2d027e0ea9c4e96bbbdfb676999a27b" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 03:50:28.617802 32427 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 03:50:30.183230 31443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:30.183401 31443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:30.186075 31443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37d0cdd28d594b4eb046ffa44aac8390: Sending DeleteTablet for 3 replicas of tablet b6a00c6e864e4242b93b214d372eea59
I20260810 03:50:30.186658 31695 tablet_service.cc:1514] Processing DeleteTablet for tablet b6a00c6e864e4242b93b214d372eea59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:30 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:46752
I20260810 03:50:30.186738 31942 tablet_service.cc:1514] Processing DeleteTablet for tablet b6a00c6e864e4242b93b214d372eea59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:30 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:59560
I20260810 03:50:30.186730 32173 tablet_service.cc:1514] Processing DeleteTablet for tablet b6a00c6e864e4242b93b214d372eea59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:30 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:60278
I20260810 03:50:30.187078 32593 tablet_replica.cc:330] T b6a00c6e864e4242b93b214d372eea59 P f2d027e0ea9c4e96bbbdfb676999a27b: stopping tablet replica
I20260810 03:50:30.187158 32599 tablet_replica.cc:330] T b6a00c6e864e4242b93b214d372eea59 P 592678774ca74107a42f086d26a5563a: stopping tablet replica
I20260810 03:50:30.187132 32592 tablet_replica.cc:330] T b6a00c6e864e4242b93b214d372eea59 P 83c344f979094f4f9323e8a15deffa5e: stopping tablet replica
I20260810 03:50:30.187158 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 31574
I20260810 03:50:30.231268 32593 raft_consensus.cc:2229] T b6a00c6e864e4242b93b214d372eea59 P f2d027e0ea9c4e96bbbdfb676999a27b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:30.231608 32593 raft_consensus.cc:2258] T b6a00c6e864e4242b93b214d372eea59 P f2d027e0ea9c4e96bbbdfb676999a27b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:30.233036 32593 ts_tablet_manager.cc:1905] T b6a00c6e864e4242b93b214d372eea59 P f2d027e0ea9c4e96bbbdfb676999a27b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:30.233477 32599 raft_consensus.cc:2229] T b6a00c6e864e4242b93b214d372eea59 P 592678774ca74107a42f086d26a5563a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:30.233651 32599 raft_consensus.cc:2258] T b6a00c6e864e4242b93b214d372eea59 P 592678774ca74107a42f086d26a5563a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:30.234404 32599 ts_tablet_manager.cc:1905] T b6a00c6e864e4242b93b214d372eea59 P 592678774ca74107a42f086d26a5563a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:30.236135 32593 ts_tablet_manager.cc:1918] T b6a00c6e864e4242b93b214d372eea59 P f2d027e0ea9c4e96bbbdfb676999a27b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 03:50:30.236228 32593 log.cc:1196] T b6a00c6e864e4242b93b214d372eea59 P f2d027e0ea9c4e96bbbdfb676999a27b: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/b6a00c6e864e4242b93b214d372eea59
I20260810 03:50:30.236483 32593 ts_tablet_manager.cc:1939] T b6a00c6e864e4242b93b214d372eea59 P f2d027e0ea9c4e96bbbdfb676999a27b: Deleting consensus metadata
I20260810 03:50:30.236893 32599 ts_tablet_manager.cc:1918] T b6a00c6e864e4242b93b214d372eea59 P 592678774ca74107a42f086d26a5563a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 03:50:30.236969 32599 log.cc:1196] T b6a00c6e864e4242b93b214d372eea59 P 592678774ca74107a42f086d26a5563a: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/b6a00c6e864e4242b93b214d372eea59
I20260810 03:50:30.237264 32599 ts_tablet_manager.cc:1939] T b6a00c6e864e4242b93b214d372eea59 P 592678774ca74107a42f086d26a5563a: Deleting consensus metadata
I20260810 03:50:30.237335 31417 catalog_manager.cc:4841] TS f2d027e0ea9c4e96bbbdfb676999a27b (127.27.153.67:37723): tablet b6a00c6e864e4242b93b214d372eea59 (table test-table [id=37c24790761949168f7e9c1ac333e7e8]) successfully deleted
I20260810 03:50:30.237959 31421 catalog_manager.cc:4841] TS 592678774ca74107a42f086d26a5563a (127.27.153.66:43007): tablet b6a00c6e864e4242b93b214d372eea59 (table test-table [id=37c24790761949168f7e9c1ac333e7e8]) successfully deleted
W20260810 03:50:30.241815 31420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.153.65:38971 (error 108)
I20260810 03:50:30.242733 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 31828
I20260810 03:50:30.258412 32601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:30.242720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:38971 (local address 127.0.0.1:46782)
0810 03:50:30.258140 (+ 15420us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:30.258258 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:38971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15314,"thread_start_us":80,"threads_started":1}
W20260810 03:50:30.258623 31420 catalog_manager.cc:4584] TS 83c344f979094f4f9323e8a15deffa5e (127.27.153.65:38971): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b6a00c6e864e4242b93b214d372eea59: Network error: Client connection negotiation failed: client connection to 127.27.153.65:38971: connect: Connection refused (error 111)
I20260810 03:50:30.265111 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 32065
I20260810 03:50:30.289170 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 31388
2026-08-10T03:50:30Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6650 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:50:30 dist-test-slave-pww0 krb5kdc[32608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:50:30 dist-test-slave-pww0 krb5kdc[32608](info): set up 2 sockets
Aug 10 03:50:30 dist-test-slave-pww0 krb5kdc[32608](info): commencing operation
krb5kdc: starting...
W20260810 03:50:31.491161 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:50:31 dist-test-slave-pww0 krb5kdc[32608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:50:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:50:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:50:31.622859 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:36485
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:41119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:36485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:31.738476 32642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:31.738806 32642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:31.738871 32642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:31.743090 32642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:50:31.743180 32642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:31.743207 32642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:31.743228 32642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:50:31.743249 32642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:50:31.749250 32642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:41119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:36485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:36485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:31.750471 32642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:31.751627 32642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:31.760900 32652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:31.760941 32659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:31.760941 32653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:31.761155 32642 server_base.cc:1034] running on GCE node
I20260810 03:50:31.761898 32642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:31.763121 32642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:31.764310 32642 hybrid_clock.cc:648] HybridClock initialized: now 1786333831764266 us; error 57 us; skew 500 ppm
Aug 10 03:50:31 dist-test-slave-pww0 krb5kdc[32608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333831, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:31.767282 32642 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:50:31.768352 32642 webserver.cc:460] Webserver started at http://127.27.153.126:38677/ using document root <none> and password file <none>
I20260810 03:50:31.768918 32642 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:31.768980 32642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:31.769162 32642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:31.771322 32642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "ba19216d06e540e0b66fc34e03f47af8"
format_stamp: "Formatted at 2026-08-10 03:50:31 on dist-test-slave-pww0"
server_key: "0498f2cbd0e2ff3e81bd8e392f072508"
server_key_iv: "5b8d84e7ba31c9c47b9c95589ec70e45"
server_key_version: "encryptionkey@0"
I20260810 03:50:31.771911 32642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "ba19216d06e540e0b66fc34e03f47af8"
format_stamp: "Formatted at 2026-08-10 03:50:31 on dist-test-slave-pww0"
server_key: "0498f2cbd0e2ff3e81bd8e392f072508"
server_key_iv: "5b8d84e7ba31c9c47b9c95589ec70e45"
server_key_version: "encryptionkey@0"
I20260810 03:50:31.775875 32642 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:50:31.778331 32675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:31.779650 32642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:50:31.779773 32642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "ba19216d06e540e0b66fc34e03f47af8"
format_stamp: "Formatted at 2026-08-10 03:50:31 on dist-test-slave-pww0"
server_key: "0498f2cbd0e2ff3e81bd8e392f072508"
server_key_iv: "5b8d84e7ba31c9c47b9c95589ec70e45"
server_key_version: "encryptionkey@0"
I20260810 03:50:31.779883 32642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:31.796422 32642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:31.799582 32642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:31.799777 32642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:31.807322 32642 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:36485
I20260810 03:50:31.807337 302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:36485 every 8 connection(s)
I20260810 03:50:31.808385 32642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:50:31.810418 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 32642
I20260810 03:50:31.810513 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:50:31.810779 28261 external_mini_cluster.cc:1428] Setting key 2eb2d8e1fac8d514ab97a413052d0f22
I20260810 03:50:31.811376 304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:31.818338 304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba19216d06e540e0b66fc34e03f47af8: Bootstrap starting.
Aug 10 03:50:31 dist-test-slave-pww0 krb5kdc[32608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:31.821106 304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba19216d06e540e0b66fc34e03f47af8: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:31.821874 304 log.cc:824] T 00000000000000000000000000000000 P ba19216d06e540e0b66fc34e03f47af8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:31.824004 304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba19216d06e540e0b66fc34e03f47af8: No bootstrap required, opened a new log
I20260810 03:50:31.825269 307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:31.812168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40524 (local address 127.27.153.126:36485)
0810 03:50:31.812613 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:31.812631 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:31.812668 (+ 37us) server_negotiation.cc:407] Connection header received
0810 03:50:31.813335 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:31.813352 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:31.813694 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:31.813987 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:31.814938 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:31.816015 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:31.817192 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:31.817677 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:31.817791 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:31.820558 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:31.820563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:31.820578 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:31.820612 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:31.822410 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:31.822887 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:31.822893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:31.822897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:31.823013 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:31.823284 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:31.823288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:31.823289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:31.823748 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:31.823949 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:31.824225 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 03:50:31.824529 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":132,"threads_started":1}
I20260810 03:50:31.827219 304 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba19216d06e540e0b66fc34e03f47af8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba19216d06e540e0b66fc34e03f47af8" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 36485 } }
I20260810 03:50:31.827497 304 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba19216d06e540e0b66fc34e03f47af8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:31.827579 304 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba19216d06e540e0b66fc34e03f47af8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba19216d06e540e0b66fc34e03f47af8, State: Initialized, Role: FOLLOWER
I20260810 03:50:31.828056 304 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba19216d06e540e0b66fc34e03f47af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba19216d06e540e0b66fc34e03f47af8" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 36485 } }
I20260810 03:50:31.828217 304 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba19216d06e540e0b66fc34e03f47af8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:50:31.828276 304 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba19216d06e540e0b66fc34e03f47af8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:50:31.828372 304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba19216d06e540e0b66fc34e03f47af8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:31.829304 304 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba19216d06e540e0b66fc34e03f47af8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba19216d06e540e0b66fc34e03f47af8" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 36485 } }
I20260810 03:50:31.829622 304 leader_election.cc:302] T 00000000000000000000000000000000 P ba19216d06e540e0b66fc34e03f47af8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba19216d06e540e0b66fc34e03f47af8; no voters:
I20260810 03:50:31.829903 304 leader_election.cc:288] T 00000000000000000000000000000000 P ba19216d06e540e0b66fc34e03f47af8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:50:31.830051 311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba19216d06e540e0b66fc34e03f47af8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:31.830307 311 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba19216d06e540e0b66fc34e03f47af8 [term 1 LEADER]: Becoming Leader. State: Replica: ba19216d06e540e0b66fc34e03f47af8, State: Running, Role: LEADER
I20260810 03:50:31.830664 311 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba19216d06e540e0b66fc34e03f47af8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba19216d06e540e0b66fc34e03f47af8" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 36485 } }
I20260810 03:50:31.831171 304 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba19216d06e540e0b66fc34e03f47af8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:50:31.832348 314 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba19216d06e540e0b66fc34e03f47af8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba19216d06e540e0b66fc34e03f47af8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba19216d06e540e0b66fc34e03f47af8" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 36485 } } }
I20260810 03:50:31.832499 314 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba19216d06e540e0b66fc34e03f47af8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:31.832844 317 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba19216d06e540e0b66fc34e03f47af8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba19216d06e540e0b66fc34e03f47af8. Latest consensus state: current_term: 1 leader_uuid: "ba19216d06e540e0b66fc34e03f47af8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba19216d06e540e0b66fc34e03f47af8" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 36485 } } }
I20260810 03:50:31.832916 332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:50:31.833521 317 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba19216d06e540e0b66fc34e03f47af8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:31.836448 332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:50:31.843156 332 catalog_manager.cc:1347] Generated new cluster ID: 00237e0b917f4b70a786641011085a78
I20260810 03:50:31.843271 332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:50:31.857769 332 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:50:31.859150 332 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:50:31.866451 332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba19216d06e540e0b66fc34e03f47af8: Generated new TSK 0
I20260810 03:50:31.867381 332 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.153.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
I20260810 03:50:31.902184 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:36485
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:41119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:32.021998 351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:32.022342 351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:32.022388 351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:32.026744 351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:32.026844 351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:32.026906 351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:50:32.033162 351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:41119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:36485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:32.034409 351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:32.035573 351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:32.046226 360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:32.046265 369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:32.046288 359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:32.046674 351 server_base.cc:1034] running on GCE node
I20260810 03:50:32.047161 351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:32.047923 351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:32.049108 351 hybrid_clock.cc:648] HybridClock initialized: now 1786333832049082 us; error 40 us; skew 500 ppm
Aug 10 03:50:32 dist-test-slave-pww0 krb5kdc[32608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333832, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:32.052270 351 init.cc:377] Logged in from keytab as kudu/127.27.153.65@KRBTEST.COM (short username kudu)
I20260810 03:50:32.053275 351 webserver.cc:460] Webserver started at http://127.27.153.65:42165/ using document root <none> and password file <none>
I20260810 03:50:32.053844 351 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:32.053905 351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:32.054087 351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:32.056396 351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "199d545bcf5d4167a30078af072706f7"
format_stamp: "Formatted at 2026-08-10 03:50:32 on dist-test-slave-pww0"
server_key: "14f9aa9b07cdb97114d714966c0b1788"
server_key_iv: "e114ce8f6b00e99d327df87e97a4c727"
server_key_version: "encryptionkey@0"
I20260810 03:50:32.056983 351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "199d545bcf5d4167a30078af072706f7"
format_stamp: "Formatted at 2026-08-10 03:50:32 on dist-test-slave-pww0"
server_key: "14f9aa9b07cdb97114d714966c0b1788"
server_key_iv: "e114ce8f6b00e99d327df87e97a4c727"
server_key_version: "encryptionkey@0"
I20260810 03:50:32.061214 351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:50:32.064018 380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:32.065493 351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:32.065618 351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "199d545bcf5d4167a30078af072706f7"
format_stamp: "Formatted at 2026-08-10 03:50:32 on dist-test-slave-pww0"
server_key: "14f9aa9b07cdb97114d714966c0b1788"
server_key_iv: "e114ce8f6b00e99d327df87e97a4c727"
server_key_version: "encryptionkey@0"
I20260810 03:50:32.065721 351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:32.085070 351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:32.088382 351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:32.088588 351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:32.089301 351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:32.090474 351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:32.090536 351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:32.090583 351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:32.090598 351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:32.099866 351 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:35143
I20260810 03:50:32.099906 603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:35143 every 8 connection(s)
I20260810 03:50:32.100869 351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:50:32.110163 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 351
I20260810 03:50:32.110314 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:50:32.110637 28261 external_mini_cluster.cc:1428] Setting key 3ed380b12de7935b3efd3ebc46213da2
Aug 10 03:50:32 dist-test-slave-pww0 krb5kdc[32608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333832, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:32.116312 307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:32.102858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:38341 (local address 127.27.153.126:36485)
0810 03:50:32.103038 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:32.103042 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:32.104050 (+ 1008us) server_negotiation.cc:407] Connection header received
0810 03:50:32.105489 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:32.105492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:32.105579 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:32.105656 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:32.106948 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:32.107568 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:32.108757 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:32.108998 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:32.109099 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:32.112539 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:32.112543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:32.112545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:32.112574 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:32.114002 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:32.114589 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:32.114593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:32.114595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:32.114689 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:32.115062 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:32.115065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:32.115067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:32.115273 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:32.115366 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:32.115981 (+ 615us) server_negotiation.cc:299] Negotiation successful
0810 03:50:32.116134 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 03:50:32.117261 608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:32.103277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:36485 (local address 127.27.153.65:38341)
0810 03:50:32.103876 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:32.103927 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:32.105259 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:32.105866 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:32.105874 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:32.106268 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:32.106786 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:32.106802 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:32.107704 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:32.107707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:32.108631 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:32.108642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:32.109162 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:32.109166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:32.109340 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:32.110009 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:32.110032 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:32.112340 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:32.114152 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:32.114160 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:32.114180 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:32.114481 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:32.114785 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:32.114788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:32.114790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:32.114965 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:32.115386 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:32.115394 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:32.115731 (+ 337us) client_negotiation.cc:769] Sending connection context
0810 03:50:32.115925 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 03:50:32.116319 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":177,"threads_started":1}
I20260810 03:50:32.118636 604 heartbeater.cc:347] Connected to a master server at 127.27.153.126:36485
I20260810 03:50:32.118884 604 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:32.119434 604 heartbeater.cc:511] Master 127.27.153.126:36485 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.153.66@KRBTEST.COM; defaulting to no policy
I20260810 03:50:32.121368 32700 ts_manager.cc:194] Registered new tserver with Master: 199d545bcf5d4167a30078af072706f7 (127.27.153.65:35143)
I20260810 03:50:32.122756 32700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:38341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
I20260810 03:50:32.142400 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:36485
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:41119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:32.261304 615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:32.261602 615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:32.261646 615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:32.265738 615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:32.265811 615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:32.265872 615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:50:32.271641 615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:41119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:36485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:32.272831 615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:32.273916 615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:32.283856 626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:32.283842 623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:32.283842 633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:32.283954 615 server_base.cc:1034] running on GCE node
I20260810 03:50:32.284612 615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:32.285320 615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:32.286521 615 hybrid_clock.cc:648] HybridClock initialized: now 1786333832286499 us; error 37 us; skew 500 ppm
Aug 10 03:50:32 dist-test-slave-pww0 krb5kdc[32608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333832, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:32.289577 615 init.cc:377] Logged in from keytab as kudu/127.27.153.66@KRBTEST.COM (short username kudu)
I20260810 03:50:32.290540 615 webserver.cc:460] Webserver started at http://127.27.153.66:46811/ using document root <none> and password file <none>
I20260810 03:50:32.291085 615 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:32.291144 615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:32.291324 615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:32.293612 615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "63de4189e60e4219818769e67d2ac40e"
format_stamp: "Formatted at 2026-08-10 03:50:32 on dist-test-slave-pww0"
server_key: "07785f114d706d192e9df64e01291c3c"
server_key_iv: "f09ddc5f4d10c3c718e67e96e2a8c996"
server_key_version: "encryptionkey@0"
I20260810 03:50:32.294186 615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "63de4189e60e4219818769e67d2ac40e"
format_stamp: "Formatted at 2026-08-10 03:50:32 on dist-test-slave-pww0"
server_key: "07785f114d706d192e9df64e01291c3c"
server_key_iv: "f09ddc5f4d10c3c718e67e96e2a8c996"
server_key_version: "encryptionkey@0"
I20260810 03:50:32.298378 615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:50:32.301318 645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:32.302778 615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:50:32.302928 615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "63de4189e60e4219818769e67d2ac40e"
format_stamp: "Formatted at 2026-08-10 03:50:32 on dist-test-slave-pww0"
server_key: "07785f114d706d192e9df64e01291c3c"
server_key_iv: "f09ddc5f4d10c3c718e67e96e2a8c996"
server_key_version: "encryptionkey@0"
I20260810 03:50:32.303038 615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:32.326676 615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:32.330226 615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:32.330442 615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:32.331068 615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:32.332183 615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:32.332247 615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:32.332300 615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:32.332321 615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:32.341540 615 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:33123
I20260810 03:50:32.341570 867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:33123 every 8 connection(s)
I20260810 03:50:32.342541 615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:50:32.350878 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 615
I20260810 03:50:32.351001 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:50:32.351264 28261 external_mini_cluster.cc:1428] Setting key 2d52753b675a473304b7dc642b033616
Aug 10 03:50:32 dist-test-slave-pww0 krb5kdc[32608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333832, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:32.358217 307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:32.344780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:39129 (local address 127.27.153.126:36485)
0810 03:50:32.344930 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:32.344934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:32.345904 (+ 970us) server_negotiation.cc:407] Connection header received
0810 03:50:32.347386 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:32.347390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:32.347497 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:32.347630 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:32.349077 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:32.349658 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:32.350986 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:32.351278 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:32.351379 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:32.354219 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:32.354223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:32.354226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:32.354257 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:32.355797 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:32.356456 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:32.356458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:32.356460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:32.356533 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:32.356908 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:32.356910 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:32.356911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:32.357117 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:32.357195 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:32.357917 (+ 722us) server_negotiation.cc:299] Negotiation successful
0810 03:50:32.358045 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 03:50:32.359484 874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:32.345194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:36485 (local address 127.27.153.66:39129)
0810 03:50:32.345721 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:32.345772 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:32.347064 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:32.347888 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:32.347897 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:32.348347 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:32.348895 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:32.348910 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:32.349804 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:32.349807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:32.350827 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:32.350838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:32.351449 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:32.351481 (+ 32us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:32.351689 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:32.352318 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:32.352337 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:32.354014 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:32.355927 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:32.355934 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:32.355956 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:32.356360 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:32.356612 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:32.356615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:32.356617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:32.356820 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:32.357219 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:32.357226 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:32.357609 (+ 383us) client_negotiation.cc:769] Sending connection context
0810 03:50:32.357853 (+ 244us) client_negotiation.cc:240] Negotiation successful
0810 03:50:32.358280 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":120,"threads_started":1}
I20260810 03:50:32.360836 869 heartbeater.cc:347] Connected to a master server at 127.27.153.126:36485
I20260810 03:50:32.361138 869 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:32.361810 869 heartbeater.cc:511] Master 127.27.153.126:36485 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.153.67@KRBTEST.COM; defaulting to no policy
I20260810 03:50:32.363194 32700 ts_manager.cc:194] Registered new tserver with Master: 63de4189e60e4219818769e67d2ac40e (127.27.153.66:33123)
I20260810 03:50:32.363883 32700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:39129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
I20260810 03:50:32.382282 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:36485
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:41119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:32.499433 881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:32.499795 881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:32.499862 881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:32.503985 881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:32.504057 881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:32.504145 881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:50:32.509651 881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:41119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:36485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:32.510821 881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:32.511927 881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:32.521940 893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:32.521960 889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:32.522250 881 server_base.cc:1034] running on GCE node
W20260810 03:50:32.521932 890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:32.522846 881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:32.523571 881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:32.524760 881 hybrid_clock.cc:648] HybridClock initialized: now 1786333832524726 us; error 50 us; skew 500 ppm
Aug 10 03:50:32 dist-test-slave-pww0 krb5kdc[32608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333832, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:32.527817 881 init.cc:377] Logged in from keytab as kudu/127.27.153.67@KRBTEST.COM (short username kudu)
I20260810 03:50:32.528832 881 webserver.cc:460] Webserver started at http://127.27.153.67:34591/ using document root <none> and password file <none>
I20260810 03:50:32.529608 881 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:32.529704 881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:32.529932 881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:32.532444 881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "c2c50ff0b67d4b0e9ebfc8fd8081cc60"
format_stamp: "Formatted at 2026-08-10 03:50:32 on dist-test-slave-pww0"
server_key: "9d2fa5978053b4691f52e793647ac6f7"
server_key_iv: "af40f54ef39af69284306cbf8441031f"
server_key_version: "encryptionkey@0"
I20260810 03:50:32.533095 881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "c2c50ff0b67d4b0e9ebfc8fd8081cc60"
format_stamp: "Formatted at 2026-08-10 03:50:32 on dist-test-slave-pww0"
server_key: "9d2fa5978053b4691f52e793647ac6f7"
server_key_iv: "af40f54ef39af69284306cbf8441031f"
server_key_version: "encryptionkey@0"
I20260810 03:50:32.537261 881 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:50:32.540089 905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:32.541589 881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:32.541742 881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "c2c50ff0b67d4b0e9ebfc8fd8081cc60"
format_stamp: "Formatted at 2026-08-10 03:50:32 on dist-test-slave-pww0"
server_key: "9d2fa5978053b4691f52e793647ac6f7"
server_key_iv: "af40f54ef39af69284306cbf8441031f"
server_key_version: "encryptionkey@0"
I20260810 03:50:32.541863 881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:32.562354 881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:32.565725 881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:32.565938 881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:32.566605 881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:32.567605 881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:32.567660 881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:32.567718 881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:32.567744 881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:32.577159 881 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:34271
I20260810 03:50:32.577183 1129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:34271 every 8 connection(s)
I20260810 03:50:32.578145 881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:50:32.580365 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 881
I20260810 03:50:32.580463 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:50:32.580724 28261 external_mini_cluster.cc:1428] Setting key b7058fbdaa799e433578cdb94e50ecdd
Aug 10 03:50:32 dist-test-slave-pww0 krb5kdc[32608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333832, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:32.592654 307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:32.580025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:40163 (local address 127.27.153.126:36485)
0810 03:50:32.580173 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:32.580178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:32.581189 (+ 1011us) server_negotiation.cc:407] Connection header received
0810 03:50:32.582567 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:32.582570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:32.582623 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:32.582704 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:32.584051 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:32.584634 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:32.585813 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:32.586015 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:32.586083 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:32.588839 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:32.588842 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:32.588844 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:32.588872 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:32.590479 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:32.591074 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:32.591077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:32.591078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:32.591162 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:32.591603 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:32.591605 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:32.591606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:32.591799 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:32.591875 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:32.592412 (+ 537us) server_negotiation.cc:299] Negotiation successful
0810 03:50:32.592523 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 03:50:32.593760 1132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:32.580489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:36485 (local address 127.27.153.67:40163)
0810 03:50:32.581004 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:32.581063 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:32.582363 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:32.582907 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:32.582915 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:32.583331 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:32.583928 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:32.583945 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:32.584768 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:32.584771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:32.585711 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:32.585720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:32.586175 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:32.586178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:32.586357 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:32.586984 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:32.587004 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:32.588638 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:32.590645 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:32.590652 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:32.590667 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:32.590976 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:32.591291 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:32.591294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:32.591295 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:32.591520 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:32.591901 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:32.591907 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:32.592199 (+ 292us) client_negotiation.cc:769] Sending connection context
0810 03:50:32.592368 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 03:50:32.592693 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":128,"threads_started":1}
I20260810 03:50:32.594961 1130 heartbeater.cc:347] Connected to a master server at 127.27.153.126:36485
I20260810 03:50:32.595209 1130 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:32.595791 1130 heartbeater.cc:511] Master 127.27.153.126:36485 requested a full tablet report, sending...
I20260810 03:50:32.596940 32700 ts_manager.cc:194] Registered new tserver with Master: c2c50ff0b67d4b0e9ebfc8fd8081cc60 (127.27.153.67:34271)
I20260810 03:50:32.597541 32700 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:40163
I20260810 03:50:32.607198 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 03:50:32.623000 307 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:50:32.621405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.27.153.126:36485)
0810 03:50:32.621595 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:32.621600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:32.621744 (+ 144us) server_negotiation.cc:407] Connection header received
0810 03:50:32.621893 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:32.621895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:32.621941 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:32.622012 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:32.622850 (+ 838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40534: BlockingRecv error: recv got EOF from 127.0.0.1:40534 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 03:50:32.624055 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 351
I20260810 03:50:32.637259 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 615
I20260810 03:50:32.650220 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 881
I20260810 03:50:32.669760 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 32642
2026-08-10T03:50:32Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2383 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:50:32 dist-test-slave-pww0 krb5kdc[1149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:50:32 dist-test-slave-pww0 krb5kdc[1149](info): set up 2 sockets
Aug 10 03:50:32 dist-test-slave-pww0 krb5kdc[1149](info): commencing operation
krb5kdc: starting...
W20260810 03:50:33.876606 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:50:33 dist-test-slave-pww0 krb5kdc[1149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:50:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:50:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:50:34.028371 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:42655
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:37425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:42655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:34.158155 1176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:34.158480 1176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:34.158527 1176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:34.162711 1176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:50:34.162806 1176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:34.162828 1176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:34.162848 1176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:50:34.162865 1176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:50:34.168519 1176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:37425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:42655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:42655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:34.169677 1176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:34.170856 1176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:34.179869 1189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:34.179875 1184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:34.179869 1183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:34.180259 1176 server_base.cc:1034] running on GCE node
I20260810 03:50:34.180953 1176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:34.182200 1176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:34.183401 1176 hybrid_clock.cc:648] HybridClock initialized: now 1786333834183373 us; error 41 us; skew 500 ppm
Aug 10 03:50:34 dist-test-slave-pww0 krb5kdc[1149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333834, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:34.186339 1176 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:50:34.187354 1176 webserver.cc:460] Webserver started at http://127.27.153.126:39963/ using document root <none> and password file <none>
I20260810 03:50:34.187990 1176 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:34.188057 1176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:34.188251 1176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:34.190447 1176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "c6bf8c8ae0d74c72a514b129aea9ed43"
format_stamp: "Formatted at 2026-08-10 03:50:34 on dist-test-slave-pww0"
server_key: "4e64c313b2ca69d9f1b040b0ae084f64"
server_key_iv: "931d154da07875e8ab898e78e39c3655"
server_key_version: "encryptionkey@0"
I20260810 03:50:34.191016 1176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "c6bf8c8ae0d74c72a514b129aea9ed43"
format_stamp: "Formatted at 2026-08-10 03:50:34 on dist-test-slave-pww0"
server_key: "4e64c313b2ca69d9f1b040b0ae084f64"
server_key_iv: "931d154da07875e8ab898e78e39c3655"
server_key_version: "encryptionkey@0"
I20260810 03:50:34.195027 1176 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260810 03:50:34.197988 1205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:34.199594 1176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:34.199733 1176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "c6bf8c8ae0d74c72a514b129aea9ed43"
format_stamp: "Formatted at 2026-08-10 03:50:34 on dist-test-slave-pww0"
server_key: "4e64c313b2ca69d9f1b040b0ae084f64"
server_key_iv: "931d154da07875e8ab898e78e39c3655"
server_key_version: "encryptionkey@0"
I20260810 03:50:34.199851 1176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:34.217861 1176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:34.221161 1176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:34.221347 1176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:34.228503 1176 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:42655
I20260810 03:50:34.228858 1303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:42655 every 8 connection(s)
I20260810 03:50:34.229588 1176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:50:34.232785 1305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:34.239091 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 1176
I20260810 03:50:34.239095 1305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43: Bootstrap starting.
I20260810 03:50:34.239233 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:50:34.239540 28261 external_mini_cluster.cc:1428] Setting key 644ee93998e043f3db9a6a9a8422654e
I20260810 03:50:34.242033 1305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:34.242925 1305 log.cc:824] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:34.245512 1305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43: No bootstrap required, opened a new log
Aug 10 03:50:34 dist-test-slave-pww0 krb5kdc[1149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:34.250278 1305 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bf8c8ae0d74c72a514b129aea9ed43" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42655 } }
I20260810 03:50:34.250609 1305 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:34.250655 1305 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bf8c8ae0d74c72a514b129aea9ed43, State: Initialized, Role: FOLLOWER
I20260810 03:50:34.251238 1305 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bf8c8ae0d74c72a514b129aea9ed43" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42655 } }
I20260810 03:50:34.251395 1305 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:50:34.251488 1305 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:50:34.251613 1305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:34.252688 1305 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bf8c8ae0d74c72a514b129aea9ed43" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42655 } }
I20260810 03:50:34.253129 1305 leader_election.cc:302] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6bf8c8ae0d74c72a514b129aea9ed43; no voters:
I20260810 03:50:34.253540 1305 leader_election.cc:288] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:50:34.253664 1315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:34.253965 1315 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43 [term 1 LEADER]: Becoming Leader. State: Replica: c6bf8c8ae0d74c72a514b129aea9ed43, State: Running, Role: LEADER
I20260810 03:50:34.254377 1315 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bf8c8ae0d74c72a514b129aea9ed43" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42655 } }
I20260810 03:50:34.255118 1305 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:50:34.256872 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:34.240923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60404 (local address 127.27.153.126:42655)
0810 03:50:34.241395 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:34.241412 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:34.241453 (+ 41us) server_negotiation.cc:407] Connection header received
0810 03:50:34.242140 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:34.242158 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:34.242560 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:34.242883 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:34.244107 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:34.245117 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:34.246409 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:34.247086 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:34.247251 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:34.250464 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:34.250471 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:34.250489 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:34.250528 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:34.252600 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:34.253204 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:34.253211 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:34.253217 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:34.253398 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:34.253778 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:34.253782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:34.253784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:34.254375 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:34.254560 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:34.255086 (+ 526us) server_negotiation.cc:299] Negotiation successful
0810 03:50:34.255578 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":115,"threads_started":1}
I20260810 03:50:34.256857 1318 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6bf8c8ae0d74c72a514b129aea9ed43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bf8c8ae0d74c72a514b129aea9ed43" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42655 } } }
I20260810 03:50:34.257300 1318 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:34.257097 1319 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6bf8c8ae0d74c72a514b129aea9ed43. Latest consensus state: current_term: 1 leader_uuid: "c6bf8c8ae0d74c72a514b129aea9ed43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bf8c8ae0d74c72a514b129aea9ed43" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42655 } } }
I20260810 03:50:34.258234 1319 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:34.260712 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260810 03:50:34.261201 1338 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:50:34.261291 1338 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:50:34.261379 1330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:50:34.262354 1330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:50:34.269438 1330 catalog_manager.cc:1347] Generated new cluster ID: b6b15a708b49464e9054f3e836b9689f
I20260810 03:50:34.269582 1330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:50:34.279114 1330 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:50:34.280155 1330 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:50:34.284353 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:34.277894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60412 (local address 127.27.153.126:42655)
0810 03:50:34.278021 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:34.278027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:34.278290 (+ 263us) server_negotiation.cc:407] Connection header received
0810 03:50:34.278415 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:34.278419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:34.278467 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:34.278548 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:34.279291 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:34.280151 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:34.281557 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:34.281925 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:34.282034 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:34.282999 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:34.283003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:34.283006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:50:34.283794 (+ 788us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:34.283835 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:50:34.283858 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:34.283953 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:34.284073 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 03:50:34.284172 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 03:50:34.290869 1330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6bf8c8ae0d74c72a514b129aea9ed43: Generated new TSK 0
I20260810 03:50:34.291729 1330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:50:34.298009 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 1176
2026-08-10T03:50:34Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1634 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:50:34 dist-test-slave-pww0 krb5kdc[1355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:50:34 dist-test-slave-pww0 krb5kdc[1355](info): set up 2 sockets
Aug 10 03:50:34 dist-test-slave-pww0 krb5kdc[1355](info): commencing operation
krb5kdc: starting...
W20260810 03:50:35.524909 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.189s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:50:35 dist-test-slave-pww0 krb5kdc[1355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:50:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:50:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:50:35.667810 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:36781
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:36781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:35.789716 1374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:35.790040 1374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:35.790100 1374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:35.794431 1374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:50:35.794521 1374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:35.794551 1374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:35.794572 1374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:50:35.794592 1374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:50:35.800413 1374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:36781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:36781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:35.801648 1374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:35.802765 1374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:35.812475 1387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:35.812666 1382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:35.812830 1374 server_base.cc:1034] running on GCE node
W20260810 03:50:35.812482 1381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:35.813482 1374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:35.814965 1374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:35.816170 1374 hybrid_clock.cc:648] HybridClock initialized: now 1786333835816150 us; error 42 us; skew 500 ppm
Aug 10 03:50:35 dist-test-slave-pww0 krb5kdc[1355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333835, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:35.826100 1374 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:50:35.827219 1374 webserver.cc:460] Webserver started at http://127.27.153.126:43517/ using document root <none> and password file <none>
I20260810 03:50:35.827914 1374 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:35.828001 1374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:35.828231 1374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:35.830731 1374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "2d992fa0a64d4139b12b41371642abb5"
format_stamp: "Formatted at 2026-08-10 03:50:35 on dist-test-slave-pww0"
server_key: "31cc60a56bdfe556180a792abb017783"
server_key_iv: "485ca9899521377aefd932a9a2a08595"
server_key_version: "encryptionkey@0"
I20260810 03:50:35.831379 1374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "2d992fa0a64d4139b12b41371642abb5"
format_stamp: "Formatted at 2026-08-10 03:50:35 on dist-test-slave-pww0"
server_key: "31cc60a56bdfe556180a792abb017783"
server_key_iv: "485ca9899521377aefd932a9a2a08595"
server_key_version: "encryptionkey@0"
I20260810 03:50:35.835790 1374 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:50:35.838779 1402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:35.840315 1374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:35.840466 1374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "2d992fa0a64d4139b12b41371642abb5"
format_stamp: "Formatted at 2026-08-10 03:50:35 on dist-test-slave-pww0"
server_key: "31cc60a56bdfe556180a792abb017783"
server_key_iv: "485ca9899521377aefd932a9a2a08595"
server_key_version: "encryptionkey@0"
I20260810 03:50:35.840601 1374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:35.853284 1374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:35.857225 1374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:35.857443 1374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:35.865229 1374 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:36781
I20260810 03:50:35.865507 1497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:36781 every 8 connection(s)
I20260810 03:50:35.866345 1374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:50:35.869699 1498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:35.876040 1498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d992fa0a64d4139b12b41371642abb5: Bootstrap starting.
I20260810 03:50:35.876613 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 1374
I20260810 03:50:35.876734 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:50:35.876997 28261 external_mini_cluster.cc:1428] Setting key 1be64a8f41f5cf7c32205300912b5da9
I20260810 03:50:35.878934 1498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d992fa0a64d4139b12b41371642abb5: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:35.879817 1498 log.cc:824] T 00000000000000000000000000000000 P 2d992fa0a64d4139b12b41371642abb5: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:35.882192 1498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d992fa0a64d4139b12b41371642abb5: No bootstrap required, opened a new log
I20260810 03:50:35.886637 1498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d992fa0a64d4139b12b41371642abb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d992fa0a64d4139b12b41371642abb5" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 36781 } }
I20260810 03:50:35.886925 1498 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d992fa0a64d4139b12b41371642abb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:35.886989 1498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d992fa0a64d4139b12b41371642abb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d992fa0a64d4139b12b41371642abb5, State: Initialized, Role: FOLLOWER
Aug 10 03:50:35 dist-test-slave-pww0 krb5kdc[1355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:35.887519 1498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d992fa0a64d4139b12b41371642abb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d992fa0a64d4139b12b41371642abb5" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 36781 } }
I20260810 03:50:35.887681 1498 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d992fa0a64d4139b12b41371642abb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:50:35.887755 1498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d992fa0a64d4139b12b41371642abb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:50:35.887859 1498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d992fa0a64d4139b12b41371642abb5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:35.889078 1498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d992fa0a64d4139b12b41371642abb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d992fa0a64d4139b12b41371642abb5" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 36781 } }
I20260810 03:50:35.889472 1498 leader_election.cc:302] T 00000000000000000000000000000000 P 2d992fa0a64d4139b12b41371642abb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d992fa0a64d4139b12b41371642abb5; no voters:
I20260810 03:50:35.889847 1498 leader_election.cc:288] T 00000000000000000000000000000000 P 2d992fa0a64d4139b12b41371642abb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:50:35.889954 1504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d992fa0a64d4139b12b41371642abb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:35.890187 1504 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d992fa0a64d4139b12b41371642abb5 [term 1 LEADER]: Becoming Leader. State: Replica: 2d992fa0a64d4139b12b41371642abb5, State: Running, Role: LEADER
I20260810 03:50:35.890630 1504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d992fa0a64d4139b12b41371642abb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d992fa0a64d4139b12b41371642abb5" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 36781 } }
I20260810 03:50:35.891328 1498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d992fa0a64d4139b12b41371642abb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:50:35.892439 1506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d992fa0a64d4139b12b41371642abb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d992fa0a64d4139b12b41371642abb5. Latest consensus state: current_term: 1 leader_uuid: "2d992fa0a64d4139b12b41371642abb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d992fa0a64d4139b12b41371642abb5" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 36781 } } }
I20260810 03:50:35.892643 1506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d992fa0a64d4139b12b41371642abb5 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:35.893007 1505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d992fa0a64d4139b12b41371642abb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d992fa0a64d4139b12b41371642abb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d992fa0a64d4139b12b41371642abb5" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 36781 } } }
I20260810 03:50:35.893178 1505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d992fa0a64d4139b12b41371642abb5 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:35.893515 1519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:50:35.893895 1502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:35.878454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43856 (local address 127.27.153.126:36781)
0810 03:50:35.878953 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:35.878971 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:35.879016 (+ 45us) server_negotiation.cc:407] Connection header received
0810 03:50:35.879812 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:35.879833 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:35.880230 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:35.880540 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:35.881303 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:35.882746 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:35.883789 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:35.884599 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:35.884790 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:35.887795 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:35.887802 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:35.887823 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:35.887866 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:35.889684 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:35.890288 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:35.890310 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:35.890318 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:35.890479 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:35.890894 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:35.890898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:35.890900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:35.891544 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:35.891753 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:35.892463 (+ 710us) server_negotiation.cc:299] Negotiation successful
0810 03:50:35.892809 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":172,"threads_started":1}
I20260810 03:50:35.897166 1519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:50:35.904011 1519 catalog_manager.cc:1347] Generated new cluster ID: fc945083936f4a5faba3b3ccfe331e68
I20260810 03:50:35.904111 1519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:50:35.918184 1519 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:50:35.919236 1519 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:50:35.925813 1519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d992fa0a64d4139b12b41371642abb5: Generated new TSK 0
I20260810 03:50:35.926607 1519 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.153.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
I20260810 03:50:35.972785 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:36781
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:36.130389 1550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:36.130719 1550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:36.130766 1550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:36.134961 1550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:36.135053 1550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:36.135119 1550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:50:36.140790 1550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:36781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:36.141953 1550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:36.143102 1550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:36.153628 1559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:36.153909 1550 server_base.cc:1034] running on GCE node
W20260810 03:50:36.154001 1560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:36.153637 1562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:36.154548 1550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:36.155301 1550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:36.156474 1550 hybrid_clock.cc:648] HybridClock initialized: now 1786333836156449 us; error 40 us; skew 500 ppm
Aug 10 03:50:36 dist-test-slave-pww0 krb5kdc[1355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333836, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:36.159832 1550 init.cc:377] Logged in from keytab as kudu/127.27.153.65@KRBTEST.COM (short username kudu)
I20260810 03:50:36.160951 1550 webserver.cc:460] Webserver started at http://127.27.153.65:41479/ using document root <none> and password file <none>
I20260810 03:50:36.161592 1550 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:36.161664 1550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:36.161970 1550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:36.164397 1550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "a171d3fd678841e2aa2989fc175eac05"
format_stamp: "Formatted at 2026-08-10 03:50:36 on dist-test-slave-pww0"
server_key: "76a3ba21f10a94b827910d3e676bfb0d"
server_key_iv: "d34cb2e81b74abae65fc26b6624f99e0"
server_key_version: "encryptionkey@0"
I20260810 03:50:36.165155 1550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "a171d3fd678841e2aa2989fc175eac05"
format_stamp: "Formatted at 2026-08-10 03:50:36 on dist-test-slave-pww0"
server_key: "76a3ba21f10a94b827910d3e676bfb0d"
server_key_iv: "d34cb2e81b74abae65fc26b6624f99e0"
server_key_version: "encryptionkey@0"
I20260810 03:50:36.169857 1550 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:50:36.172825 1572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:36.174302 1550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:36.174501 1550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "a171d3fd678841e2aa2989fc175eac05"
format_stamp: "Formatted at 2026-08-10 03:50:36 on dist-test-slave-pww0"
server_key: "76a3ba21f10a94b827910d3e676bfb0d"
server_key_iv: "d34cb2e81b74abae65fc26b6624f99e0"
server_key_version: "encryptionkey@0"
I20260810 03:50:36.174630 1550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:36.201529 1550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:36.205044 1550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:36.205257 1550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:36.205928 1550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:36.206869 1550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:36.206921 1550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:36.206967 1550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:36.207000 1550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:36.216403 1550 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:43965
I20260810 03:50:36.216490 1767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:43965 every 8 connection(s)
I20260810 03:50:36.217396 1550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:50:36.224161 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 1550
I20260810 03:50:36.224300 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:50:36.224575 28261 external_mini_cluster.cc:1428] Setting key 5c89900bdb20be920dbb27144d41d127
Aug 10 03:50:36 dist-test-slave-pww0 krb5kdc[1355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333836, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:36.233278 1502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:36.219328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:59873 (local address 127.27.153.126:36781)
0810 03:50:36.219518 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:36.219524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:36.220494 (+ 970us) server_negotiation.cc:407] Connection header received
0810 03:50:36.222039 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:36.222042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:36.222095 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:36.222183 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:36.223532 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:36.224132 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:36.225512 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:36.225797 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:36.225896 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:36.229048 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:36.229052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:36.229054 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:36.229102 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:36.230663 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:36.231407 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:36.231411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:36.231412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:36.231540 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:36.231986 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:36.231989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:36.231990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:36.232194 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:36.232269 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:36.232989 (+ 720us) server_negotiation.cc:299] Negotiation successful
0810 03:50:36.233137 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":64}
I20260810 03:50:36.234578 1773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:36.219773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:36781 (local address 127.27.153.65:59873)
0810 03:50:36.220322 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:36.220374 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:36.221787 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:36.222359 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:36.222366 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:36.222788 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:36.223314 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:36.223331 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:36.224292 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:36.224296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:36.225382 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:36.225393 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:36.225992 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:36.225996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:36.226214 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:36.226839 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:36.226858 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:36.228813 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:36.230823 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:36.230831 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:36.230850 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:36.231276 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:36.231665 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:36.231669 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:36.231671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:36.231890 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:36.232296 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:36.232305 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:36.232709 (+ 404us) client_negotiation.cc:769] Sending connection context
0810 03:50:36.232965 (+ 256us) client_negotiation.cc:240] Negotiation successful
0810 03:50:36.233378 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":147,"threads_started":1}
WARNING: no policy specified for kudu/127.27.153.66@KRBTEST.COM; defaulting to no policy
I20260810 03:50:36.236112 1768 heartbeater.cc:347] Connected to a master server at 127.27.153.126:36781
I20260810 03:50:36.236377 1768 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:36.236948 1768 heartbeater.cc:511] Master 127.27.153.126:36781 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.66@KRBTEST.COM" created.
I20260810 03:50:36.238790 1432 ts_manager.cc:194] Registered new tserver with Master: a171d3fd678841e2aa2989fc175eac05 (127.27.153.65:43965)
I20260810 03:50:36.240372 1432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:59873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
I20260810 03:50:36.260061 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:36781
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:36.381744 1778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:36.382063 1778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:36.382125 1778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:36.386341 1778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:36.386426 1778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:36.386511 1778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:50:36.392469 1778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:36781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:36.393720 1778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:36.394791 1778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:36.405194 1796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:36.405194 1791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:36.405259 1778 server_base.cc:1034] running on GCE node
W20260810 03:50:36.405200 1792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:36.405859 1778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:36.406543 1778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:36.407703 1778 hybrid_clock.cc:648] HybridClock initialized: now 1786333836407682 us; error 34 us; skew 500 ppm
Aug 10 03:50:36 dist-test-slave-pww0 krb5kdc[1355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333836, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:36.410579 1778 init.cc:377] Logged in from keytab as kudu/127.27.153.66@KRBTEST.COM (short username kudu)
I20260810 03:50:36.411576 1778 webserver.cc:460] Webserver started at http://127.27.153.66:41649/ using document root <none> and password file <none>
I20260810 03:50:36.412122 1778 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:36.412180 1778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:36.412357 1778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:36.414466 1778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "22b1dd22ca2a4014aa2b500969aefa82"
format_stamp: "Formatted at 2026-08-10 03:50:36 on dist-test-slave-pww0"
server_key: "66f0172ead5a2c3ac034a49377539cf4"
server_key_iv: "ecbd09d281ff4bee55a3c98c315677ac"
server_key_version: "encryptionkey@0"
I20260810 03:50:36.415007 1778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "22b1dd22ca2a4014aa2b500969aefa82"
format_stamp: "Formatted at 2026-08-10 03:50:36 on dist-test-slave-pww0"
server_key: "66f0172ead5a2c3ac034a49377539cf4"
server_key_iv: "ecbd09d281ff4bee55a3c98c315677ac"
server_key_version: "encryptionkey@0"
I20260810 03:50:36.419045 1778 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260810 03:50:36.421785 1807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:36.423166 1778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:50:36.423277 1778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "22b1dd22ca2a4014aa2b500969aefa82"
format_stamp: "Formatted at 2026-08-10 03:50:36 on dist-test-slave-pww0"
server_key: "66f0172ead5a2c3ac034a49377539cf4"
server_key_iv: "ecbd09d281ff4bee55a3c98c315677ac"
server_key_version: "encryptionkey@0"
I20260810 03:50:36.423377 1778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:36.442247 1778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:36.445735 1778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:36.445950 1778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:36.446611 1778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:36.447546 1778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:36.447602 1778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:36.447651 1778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:36.447679 1778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:36.456585 1778 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:38081
I20260810 03:50:36.456679 2012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:38081 every 8 connection(s)
I20260810 03:50:36.457537 1778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:50:36.458252 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 1778
I20260810 03:50:36.458370 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:50:36.458638 28261 external_mini_cluster.cc:1428] Setting key 4cda3d0487700610ea1e8eb95d79b6de
Aug 10 03:50:36 dist-test-slave-pww0 krb5kdc[1355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333836, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.153.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.67@KRBTEST.COM" created.
I20260810 03:50:36.473178 1502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:36.459491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:35375 (local address 127.27.153.126:36781)
0810 03:50:36.459630 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:36.459636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:36.460679 (+ 1043us) server_negotiation.cc:407] Connection header received
0810 03:50:36.462083 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:36.462087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:36.462155 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:36.462230 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:36.463557 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:36.464335 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:36.465739 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:36.466055 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:36.466159 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:36.469096 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:36.469100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:36.469104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:36.469140 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:36.470675 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:36.471273 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:36.471276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:36.471277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:36.471369 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:36.471881 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:36.471884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:36.471886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:36.472131 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:36.472237 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:36.472850 (+ 613us) server_negotiation.cc:299] Negotiation successful
0810 03:50:36.473004 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 03:50:36.474092 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:36.459999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:36781 (local address 127.27.153.66:35375)
0810 03:50:36.460507 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:36.460552 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:36.461860 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:36.462418 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:36.462426 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:36.462824 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:36.463378 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:36.463393 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:36.464516 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:36.464519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:36.465579 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:36.465592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:36.466270 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:36.466275 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:36.466455 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:36.467184 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:36.467204 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:36.468859 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:36.470817 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:36.470824 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:36.470841 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:36.471142 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:36.471540 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:36.471543 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:36.471545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:36.471748 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:36.472266 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:36.472272 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:36.472563 (+ 291us) client_negotiation.cc:769] Sending connection context
0810 03:50:36.472758 (+ 195us) client_negotiation.cc:240] Negotiation successful
0810 03:50:36.473165 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":148,"threads_started":1}
I20260810 03:50:36.475863 2013 heartbeater.cc:347] Connected to a master server at 127.27.153.126:36781
I20260810 03:50:36.476172 2013 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:36.476816 2013 heartbeater.cc:511] Master 127.27.153.126:36781 requested a full tablet report, sending...
I20260810 03:50:36.478209 1432 ts_manager.cc:194] Registered new tserver with Master: 22b1dd22ca2a4014aa2b500969aefa82 (127.27.153.66:38081)
I20260810 03:50:36.478914 1432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:35375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
I20260810 03:50:36.491250 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:36781
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:37523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:36.609987 2022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:36.610287 2022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:36.610349 2022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:36.614544 2022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:36.614626 2022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:36.614693 2022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:50:36.620347 2022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:37523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:36781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:36.621510 2022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:36.622620 2022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:36.633462 2037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:36.633538 2022 server_base.cc:1034] running on GCE node
W20260810 03:50:36.633471 2033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:36.633471 2031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:36.634150 2022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:36.634891 2022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:36.636060 2022 hybrid_clock.cc:648] HybridClock initialized: now 1786333836636034 us; error 38 us; skew 500 ppm
Aug 10 03:50:36 dist-test-slave-pww0 krb5kdc[1355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333836, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:36.639148 2022 init.cc:377] Logged in from keytab as kudu/127.27.153.67@KRBTEST.COM (short username kudu)
I20260810 03:50:36.640245 2022 webserver.cc:460] Webserver started at http://127.27.153.67:33839/ using document root <none> and password file <none>
I20260810 03:50:36.640848 2022 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:36.640918 2022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:36.641134 2022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:36.643497 2022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "e7a501bf965f4718919d797f15a16df2"
format_stamp: "Formatted at 2026-08-10 03:50:36 on dist-test-slave-pww0"
server_key: "d7d6afd22986854d5354025019e12760"
server_key_iv: "394b9f8b1eff4fea1c31f9aa3c1f09a8"
server_key_version: "encryptionkey@0"
I20260810 03:50:36.644125 2022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "e7a501bf965f4718919d797f15a16df2"
format_stamp: "Formatted at 2026-08-10 03:50:36 on dist-test-slave-pww0"
server_key: "d7d6afd22986854d5354025019e12760"
server_key_iv: "394b9f8b1eff4fea1c31f9aa3c1f09a8"
server_key_version: "encryptionkey@0"
I20260810 03:50:36.648412 2022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:50:36.651361 2049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:36.652942 2022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:36.653107 2022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "e7a501bf965f4718919d797f15a16df2"
format_stamp: "Formatted at 2026-08-10 03:50:36 on dist-test-slave-pww0"
server_key: "d7d6afd22986854d5354025019e12760"
server_key_iv: "394b9f8b1eff4fea1c31f9aa3c1f09a8"
server_key_version: "encryptionkey@0"
I20260810 03:50:36.653246 2022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:36.675688 2022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:36.679440 2022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:36.679751 2022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:36.680390 2022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:36.681330 2022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:36.681382 2022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:36.681425 2022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:36.681440 2022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:36.691181 2022 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:35007
I20260810 03:50:36.691198 2262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:35007 every 8 connection(s)
I20260810 03:50:36.692293 2022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:50:36.699611 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 2022
I20260810 03:50:36.699774 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:50:36.700105 28261 external_mini_cluster.cc:1428] Setting key fdfc85f803acaf67797e287a33cb0d4a
Aug 10 03:50:36 dist-test-slave-pww0 krb5kdc[1355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333836, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:36.708070 1502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:36.694448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:60777 (local address 127.27.153.126:36781)
0810 03:50:36.694610 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:36.694615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:36.695606 (+ 991us) server_negotiation.cc:407] Connection header received
0810 03:50:36.697105 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:36.697108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:36.697169 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:36.697260 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:36.698714 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:36.699274 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:36.700880 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:36.701193 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:36.701314 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:36.704137 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:36.704141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:36.704143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:36.704171 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:36.705791 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:36.706419 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:36.706422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:36.706424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:36.706511 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:36.706871 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:36.706875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:36.706876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:36.707075 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:36.707162 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:36.707762 (+ 600us) server_negotiation.cc:299] Negotiation successful
0810 03:50:36.707903 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:50:36.709017 2268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:36.694893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:36781 (local address 127.27.153.67:60777)
0810 03:50:36.695412 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:36.695493 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:36.696833 (+ 1340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:36.697503 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:36.697511 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:36.698000 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:36.698529 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:36.698544 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:36.699433 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:36.699436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:36.700755 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:36.700767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:36.701372 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:36.701376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:36.701538 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:36.702189 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:36.702211 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:36.703948 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:36.705942 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:36.705949 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:36.705966 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:36.706296 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:36.706601 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:36.706604 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:36.706606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:36.706768 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:36.707185 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:36.707193 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:36.707503 (+ 310us) client_negotiation.cc:769] Sending connection context
0810 03:50:36.707691 (+ 188us) client_negotiation.cc:240] Negotiation successful
0810 03:50:36.708058 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":138,"threads_started":1}
I20260810 03:50:36.710407 2265 heartbeater.cc:347] Connected to a master server at 127.27.153.126:36781
I20260810 03:50:36.710733 2265 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:36.711323 2265 heartbeater.cc:511] Master 127.27.153.126:36781 requested a full tablet report, sending...
I20260810 03:50:36.712826 1432 ts_manager.cc:194] Registered new tserver with Master: e7a501bf965f4718919d797f15a16df2 (127.27.153.67:35007)
I20260810 03:50:36.713446 1432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:60777
I20260810 03:50:36.715057 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 03:50:36 dist-test-slave-pww0 krb5kdc[1355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333836, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 03:50:36 dist-test-slave-pww0 krb5kdc[1355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333836, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:36.745990 1502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:36.734782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43864 (local address 127.27.153.126:36781)
0810 03:50:36.734947 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:36.734953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:36.735102 (+ 149us) server_negotiation.cc:407] Connection header received
0810 03:50:36.735284 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:36.735287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:36.735336 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:36.735435 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:36.736260 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:36.736812 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:36.737753 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:36.738015 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:36.738125 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:36.741305 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:36.741309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:36.741312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:36.741352 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:36.742865 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:36.743558 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:36.743562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:36.743563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:36.743676 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:36.744104 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:36.744107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:36.744161 (+ 54us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:36.744485 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:36.745151 (+ 666us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:36.745639 (+ 488us) server_negotiation.cc:299] Negotiation successful
0810 03:50:36.745798 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 03:50:36.746455 1432 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43864
I20260810 03:50:36.747722 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 1550
I20260810 03:50:36.759303 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 1778
I20260810 03:50:36.777459 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 2022
I20260810 03:50:36.791008 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 1374
2026-08-10T03:50:36Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2484 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:50:36 dist-test-slave-pww0 krb5kdc[2293](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:50:36 dist-test-slave-pww0 krb5kdc[2293](info): set up 2 sockets
Aug 10 03:50:36 dist-test-slave-pww0 krb5kdc[2293](info): commencing operation
krb5kdc: starting...
W20260810 03:50:38.025087 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:50:38 dist-test-slave-pww0 krb5kdc[2293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:50:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:50:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:50:38.160449 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:44675
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:44675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:38.282986 2319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:38.283291 2319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:38.283335 2319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:38.287575 2319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:50:38.287660 2319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:38.287689 2319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:38.287719 2319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:50:38.287742 2319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:50:38.293970 2319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:44675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:44675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:38.295277 2319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:38.296475 2319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:38.305955 2334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:38.305958 2333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:38.305981 2337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:38.306257 2319 server_base.cc:1034] running on GCE node
I20260810 03:50:38.307129 2319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:38.308655 2319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:38.309857 2319 hybrid_clock.cc:648] HybridClock initialized: now 1786333838309835 us; error 41 us; skew 500 ppm
Aug 10 03:50:38 dist-test-slave-pww0 krb5kdc[2293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333838, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:38.313107 2319 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:50:38.314113 2319 webserver.cc:460] Webserver started at http://127.27.153.126:37723/ using document root <none> and password file <none>
I20260810 03:50:38.314690 2319 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:38.314752 2319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:38.314926 2319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:38.317207 2319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "5b84c8b9608843dca2b8cd0ed0a0e252"
format_stamp: "Formatted at 2026-08-10 03:50:38 on dist-test-slave-pww0"
server_key: "3cccb752c571b8a9ef56f11579babd1f"
server_key_iv: "7c96894d05ebc4643be153ef6a79f52e"
server_key_version: "encryptionkey@0"
I20260810 03:50:38.317809 2319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "5b84c8b9608843dca2b8cd0ed0a0e252"
format_stamp: "Formatted at 2026-08-10 03:50:38 on dist-test-slave-pww0"
server_key: "3cccb752c571b8a9ef56f11579babd1f"
server_key_iv: "7c96894d05ebc4643be153ef6a79f52e"
server_key_version: "encryptionkey@0"
I20260810 03:50:38.322405 2319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 03:50:38.325225 2347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:38.326896 2319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:50:38.327037 2319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "5b84c8b9608843dca2b8cd0ed0a0e252"
format_stamp: "Formatted at 2026-08-10 03:50:38 on dist-test-slave-pww0"
server_key: "3cccb752c571b8a9ef56f11579babd1f"
server_key_iv: "7c96894d05ebc4643be153ef6a79f52e"
server_key_version: "encryptionkey@0"
I20260810 03:50:38.327190 2319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:38.343593 2319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:38.347432 2319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:38.347699 2319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:38.356407 2319 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:44675
I20260810 03:50:38.356462 2444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:44675 every 8 connection(s)
I20260810 03:50:38.357491 2319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:50:38.359100 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 2319
I20260810 03:50:38.359295 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:50:38.359797 28261 external_mini_cluster.cc:1428] Setting key 16e69d78ef5b9283c57cdb3f53909735
I20260810 03:50:38.360993 2447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:38.368897 2447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252: Bootstrap starting.
Aug 10 03:50:38 dist-test-slave-pww0 krb5kdc[2293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:38.371630 2447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:38.372634 2447 log.cc:824] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:38.374969 2447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252: No bootstrap required, opened a new log
I20260810 03:50:38.376346 2457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:38.361283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43028 (local address 127.27.153.126:44675)
0810 03:50:38.361798 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:38.361820 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:38.361866 (+ 46us) server_negotiation.cc:407] Connection header received
0810 03:50:38.362754 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:38.362779 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:38.363256 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:38.363709 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:38.364479 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:38.366170 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:38.367190 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:38.367996 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:38.368174 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:38.371057 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:38.371065 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:38.371084 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:38.371127 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:38.373285 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:38.373778 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:38.373785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:38.373790 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:38.373915 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:38.374215 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:38.374219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:38.374220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:38.374673 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:38.374855 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:38.375195 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 03:50:38.375552 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":149,"threads_started":1}
I20260810 03:50:38.379331 2447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b84c8b9608843dca2b8cd0ed0a0e252" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 44675 } }
I20260810 03:50:38.379635 2447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:38.379711 2447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b84c8b9608843dca2b8cd0ed0a0e252, State: Initialized, Role: FOLLOWER
I20260810 03:50:38.380148 2447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b84c8b9608843dca2b8cd0ed0a0e252" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 44675 } }
I20260810 03:50:38.380286 2447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:50:38.380354 2447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:50:38.380455 2447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:38.381488 2447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b84c8b9608843dca2b8cd0ed0a0e252" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 44675 } }
I20260810 03:50:38.381958 2447 leader_election.cc:302] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b84c8b9608843dca2b8cd0ed0a0e252; no voters:
I20260810 03:50:38.382380 2447 leader_election.cc:288] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:50:38.382720 2462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:38.383149 2462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252 [term 1 LEADER]: Becoming Leader. State: Replica: 5b84c8b9608843dca2b8cd0ed0a0e252, State: Running, Role: LEADER
I20260810 03:50:38.383657 2462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b84c8b9608843dca2b8cd0ed0a0e252" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 44675 } }
I20260810 03:50:38.384052 2447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:50:38.386154 2466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b84c8b9608843dca2b8cd0ed0a0e252. Latest consensus state: current_term: 1 leader_uuid: "5b84c8b9608843dca2b8cd0ed0a0e252" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b84c8b9608843dca2b8cd0ed0a0e252" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 44675 } } }
I20260810 03:50:38.386121 2463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b84c8b9608843dca2b8cd0ed0a0e252" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b84c8b9608843dca2b8cd0ed0a0e252" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 44675 } } }
I20260810 03:50:38.386307 2466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:38.386315 2463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252 [sys.catalog]: This master's current role is: LEADER
W20260810 03:50:38.390071 2489 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:50:38.390189 2489 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:50:38.390636 2490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:50:38.391500 2490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:50:38.397766 2490 catalog_manager.cc:1347] Generated new cluster ID: 8e05d5df8bc04569a2953ed0ccf01004
I20260810 03:50:38.397967 2490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:50:38.408022 2490 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:50:38.409085 2490 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:50:38.414144 2490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b84c8b9608843dca2b8cd0ed0a0e252: Generated new TSK 0
I20260810 03:50:38.414932 2490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.153.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
I20260810 03:50:38.457686 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:44675
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:38.583055 2499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:38.583382 2499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:38.583428 2499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:38.588342 2499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:38.588429 2499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:38.588492 2499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:50:38.594193 2499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:44675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:38.595407 2499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:38.596513 2499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:38.607303 2514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:38.607394 2511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:38.607352 2510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:38.607589 2499 server_base.cc:1034] running on GCE node
I20260810 03:50:38.608047 2499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:38.608841 2499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:38.610037 2499 hybrid_clock.cc:648] HybridClock initialized: now 1786333838609998 us; error 54 us; skew 500 ppm
Aug 10 03:50:38 dist-test-slave-pww0 krb5kdc[2293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333838, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:38.613190 2499 init.cc:377] Logged in from keytab as kudu/127.27.153.65@KRBTEST.COM (short username kudu)
I20260810 03:50:38.614224 2499 webserver.cc:460] Webserver started at http://127.27.153.65:36437/ using document root <none> and password file <none>
I20260810 03:50:38.614849 2499 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:38.614923 2499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:38.615198 2499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:38.617620 2499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "58bb4ff23c8c4c229f7f3c4af48e1a78"
format_stamp: "Formatted at 2026-08-10 03:50:38 on dist-test-slave-pww0"
server_key: "4902f11f9975e1b937b679bada9066b5"
server_key_iv: "9ca3a1d281509dd1e01ea55ec6d60f43"
server_key_version: "encryptionkey@0"
I20260810 03:50:38.618250 2499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "58bb4ff23c8c4c229f7f3c4af48e1a78"
format_stamp: "Formatted at 2026-08-10 03:50:38 on dist-test-slave-pww0"
server_key: "4902f11f9975e1b937b679bada9066b5"
server_key_iv: "9ca3a1d281509dd1e01ea55ec6d60f43"
server_key_version: "encryptionkey@0"
I20260810 03:50:38.622537 2499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:50:38.625305 2522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:38.626674 2499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:50:38.626820 2499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "58bb4ff23c8c4c229f7f3c4af48e1a78"
format_stamp: "Formatted at 2026-08-10 03:50:38 on dist-test-slave-pww0"
server_key: "4902f11f9975e1b937b679bada9066b5"
server_key_iv: "9ca3a1d281509dd1e01ea55ec6d60f43"
server_key_version: "encryptionkey@0"
I20260810 03:50:38.626932 2499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:38.644222 2499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:38.647856 2499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:38.648041 2499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:38.648643 2499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:38.649544 2499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:38.649595 2499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:38.649637 2499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:38.649658 2499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:38.658959 2499 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:38623
I20260810 03:50:38.658987 2721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:38623 every 8 connection(s)
I20260810 03:50:38.659965 2499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:50:38.666326 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 2499
I20260810 03:50:38.666473 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:50:38.666801 28261 external_mini_cluster.cc:1428] Setting key 6328db35b35fcb931d9c5390f0ba4c9f
Aug 10 03:50:38 dist-test-slave-pww0 krb5kdc[2293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333838, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:38.675194 2457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:38.661901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:59843 (local address 127.27.153.126:44675)
0810 03:50:38.662061 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:38.662064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:38.662962 (+ 898us) server_negotiation.cc:407] Connection header received
0810 03:50:38.664472 (+ 1510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:38.664475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:38.664542 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:38.664615 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:38.665945 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:38.666684 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:38.668126 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:38.668363 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:38.668456 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:38.671257 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:38.671262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:38.671265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:38.671302 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:38.672824 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:38.673405 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:38.673409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:38.673411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:38.673517 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:38.673892 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:38.673896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:38.673898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:38.674155 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:38.674258 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:38.674833 (+ 575us) server_negotiation.cc:299] Negotiation successful
0810 03:50:38.674994 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 03:50:38.676088 2727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:38.662290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:44675 (local address 127.27.153.65:59843)
0810 03:50:38.662796 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:38.662845 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:38.664220 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:38.664821 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:38.664828 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:38.665218 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:38.665774 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:38.665790 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:38.666836 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:38.666839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:38.667990 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:38.668003 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:38.668523 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:38.668527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:38.668708 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:38.669329 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:38.669348 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:38.671045 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:38.672955 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:38.672962 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:38.672977 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:38.673294 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:38.673628 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:38.673631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:38.673632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:38.673798 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:38.674284 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:38.674289 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:38.674577 (+ 288us) client_negotiation.cc:769] Sending connection context
0810 03:50:38.674761 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 03:50:38.675091 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":119,"threads_started":1}
I20260810 03:50:38.677503 2722 heartbeater.cc:347] Connected to a master server at 127.27.153.126:44675
WARNING: no policy specified for kudu/127.27.153.66@KRBTEST.COM; defaulting to no policy
I20260810 03:50:38.677816 2722 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:38.678411 2722 heartbeater.cc:511] Master 127.27.153.126:44675 requested a full tablet report, sending...
I20260810 03:50:38.680272 2378 ts_manager.cc:194] Registered new tserver with Master: 58bb4ff23c8c4c229f7f3c4af48e1a78 (127.27.153.65:38623)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.66@KRBTEST.COM" created.
I20260810 03:50:38.681618 2378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:59843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
I20260810 03:50:38.700209 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:44675
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:38.825243 2735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:38.825542 2735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:38.825587 2735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:38.829778 2735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:38.829849 2735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:38.829908 2735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:50:38.835621 2735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:44675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:38.836774 2735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:38.837805 2735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:38.848099 2742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:38.848109 2743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:38.848183 2735 server_base.cc:1034] running on GCE node
W20260810 03:50:38.848109 2750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:38.848837 2735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:38.849586 2735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:38.850764 2735 hybrid_clock.cc:648] HybridClock initialized: now 1786333838850740 us; error 34 us; skew 500 ppm
Aug 10 03:50:38 dist-test-slave-pww0 krb5kdc[2293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333838, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:38.853652 2735 init.cc:377] Logged in from keytab as kudu/127.27.153.66@KRBTEST.COM (short username kudu)
I20260810 03:50:38.854655 2735 webserver.cc:460] Webserver started at http://127.27.153.66:35695/ using document root <none> and password file <none>
I20260810 03:50:38.855239 2735 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:38.855304 2735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:38.855525 2735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:38.857663 2735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "8c5f4f64e20c4266b78dcd1886028d51"
format_stamp: "Formatted at 2026-08-10 03:50:38 on dist-test-slave-pww0"
server_key: "7fc920445ad6080af158f66897411123"
server_key_iv: "ac2abd2881f5c3f9694f819582fa7e99"
server_key_version: "encryptionkey@0"
I20260810 03:50:38.858228 2735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "8c5f4f64e20c4266b78dcd1886028d51"
format_stamp: "Formatted at 2026-08-10 03:50:38 on dist-test-slave-pww0"
server_key: "7fc920445ad6080af158f66897411123"
server_key_iv: "ac2abd2881f5c3f9694f819582fa7e99"
server_key_version: "encryptionkey@0"
I20260810 03:50:38.862236 2735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260810 03:50:38.865136 2761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:38.866528 2735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:50:38.866679 2735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "8c5f4f64e20c4266b78dcd1886028d51"
format_stamp: "Formatted at 2026-08-10 03:50:38 on dist-test-slave-pww0"
server_key: "7fc920445ad6080af158f66897411123"
server_key_iv: "ac2abd2881f5c3f9694f819582fa7e99"
server_key_version: "encryptionkey@0"
I20260810 03:50:38.866815 2735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:38.887107 2735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:38.890695 2735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:38.890985 2735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:38.891734 2735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:38.892690 2735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:38.892745 2735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:38.892807 2735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:38.892830 2735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:38.901980 2735 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:46127
I20260810 03:50:38.902010 2969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:46127 every 8 connection(s)
I20260810 03:50:38.902992 2735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:50:38.910266 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 2735
I20260810 03:50:38.910405 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:50:38.910669 28261 external_mini_cluster.cc:1428] Setting key 55e30a6e70fc2220db72dc42bd6b3b09
Aug 10 03:50:38 dist-test-slave-pww0 krb5kdc[2293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333838, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:38.918944 2457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:38.905051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:47757 (local address 127.27.153.126:44675)
0810 03:50:38.905225 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:38.905230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:38.906254 (+ 1024us) server_negotiation.cc:407] Connection header received
0810 03:50:38.907865 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:38.907869 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:38.907926 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:38.908011 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:38.909255 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:38.909795 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:38.911310 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:38.911620 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:38.911739 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:38.914808 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:38.914811 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:38.914813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:38.914843 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:38.916425 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:38.917093 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:38.917097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:38.917099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:38.917197 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:38.917631 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:38.917635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:38.917637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:38.917915 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:38.918022 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:38.918607 (+ 585us) server_negotiation.cc:299] Negotiation successful
0810 03:50:38.918751 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 03:50:38.919907 2972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:38.905476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:44675 (local address 127.27.153.66:47757)
0810 03:50:38.906054 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:38.906122 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:38.907600 (+ 1478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:38.908176 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:38.908183 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:38.908575 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:38.909119 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:38.909134 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:38.909962 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:38.909966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:38.911153 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:38.911166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:38.911796 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:38.911800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:38.912007 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:38.912699 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:38.912718 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:38.914518 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:38.916593 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:38.916604 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:38.916620 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:38.916971 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:38.917337 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:38.917340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:38.917342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:38.917519 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:38.918027 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:38.918033 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:38.918332 (+ 299us) client_negotiation.cc:769] Sending connection context
0810 03:50:38.918513 (+ 181us) client_negotiation.cc:240] Negotiation successful
0810 03:50:38.918842 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":128,"threads_started":1}
I20260810 03:50:38.921162 2970 heartbeater.cc:347] Connected to a master server at 127.27.153.126:44675
I20260810 03:50:38.921465 2970 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.153.67@KRBTEST.COM; defaulting to no policy
I20260810 03:50:38.922187 2970 heartbeater.cc:511] Master 127.27.153.126:44675 requested a full tablet report, sending...
I20260810 03:50:38.923384 2378 ts_manager.cc:194] Registered new tserver with Master: 8c5f4f64e20c4266b78dcd1886028d51 (127.27.153.66:46127)
I20260810 03:50:38.924012 2378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:47757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
I20260810 03:50:38.943166 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:44675
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:45515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:39.061607 2976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:39.061902 2976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:39.061946 2976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:39.066140 2976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:39.066224 2976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:39.066286 2976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:50:39.071985 2976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:45515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:44675
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:39.073094 2976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:39.074112 2976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:39.084219 2987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:39.084281 2976 server_base.cc:1034] running on GCE node
W20260810 03:50:39.084250 2989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:39.084218 2993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:39.084980 2976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:39.085727 2976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:39.086913 2976 hybrid_clock.cc:648] HybridClock initialized: now 1786333839086882 us; error 46 us; skew 500 ppm
Aug 10 03:50:39 dist-test-slave-pww0 krb5kdc[2293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333839, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:39.089872 2976 init.cc:377] Logged in from keytab as kudu/127.27.153.67@KRBTEST.COM (short username kudu)
I20260810 03:50:39.090878 2976 webserver.cc:460] Webserver started at http://127.27.153.67:37115/ using document root <none> and password file <none>
I20260810 03:50:39.091434 2976 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:39.091544 2976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:39.091784 2976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:39.094129 2976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "d34d1d3504474fa5a3bd794d2bee5da3"
format_stamp: "Formatted at 2026-08-10 03:50:39 on dist-test-slave-pww0"
server_key: "c6b2fdf4db6e7d756ffeb333b2716d25"
server_key_iv: "d863fc78c4629604fa7dff047aa1d4e7"
server_key_version: "encryptionkey@0"
I20260810 03:50:39.094748 2976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "d34d1d3504474fa5a3bd794d2bee5da3"
format_stamp: "Formatted at 2026-08-10 03:50:39 on dist-test-slave-pww0"
server_key: "c6b2fdf4db6e7d756ffeb333b2716d25"
server_key_iv: "d863fc78c4629604fa7dff047aa1d4e7"
server_key_version: "encryptionkey@0"
I20260810 03:50:39.099010 2976 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:50:39.101835 3009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:39.103231 2976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:50:39.103350 2976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "d34d1d3504474fa5a3bd794d2bee5da3"
format_stamp: "Formatted at 2026-08-10 03:50:39 on dist-test-slave-pww0"
server_key: "c6b2fdf4db6e7d756ffeb333b2716d25"
server_key_iv: "d863fc78c4629604fa7dff047aa1d4e7"
server_key_version: "encryptionkey@0"
I20260810 03:50:39.103492 2976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:39.125941 2976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:39.129819 2976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:39.130021 2976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:39.130875 2976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:39.132128 2976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:39.132192 2976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:39.132253 2976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:39.132283 2976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:39.141603 2976 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:41199
I20260810 03:50:39.141629 3198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:41199 every 8 connection(s)
I20260810 03:50:39.142599 2976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:50:39.151376 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 2976
I20260810 03:50:39.151531 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:50:39.151846 28261 external_mini_cluster.cc:1428] Setting key ec98d7def144575f45d49919985b470f
Aug 10 03:50:39 dist-test-slave-pww0 krb5kdc[2293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333839, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:39.157625 2457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:39.144750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:33939 (local address 127.27.153.126:44675)
0810 03:50:39.144884 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:39.144888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:39.145833 (+ 945us) server_negotiation.cc:407] Connection header received
0810 03:50:39.147225 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:39.147228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:39.147283 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:39.147365 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:39.148672 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:39.149222 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:39.150491 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:39.150754 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:39.150849 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:39.153738 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:39.153742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:39.153745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:39.153777 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:39.155263 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:39.155929 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:39.155932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:39.155933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:39.156029 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:39.156487 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:39.156491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:39.156492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:39.156745 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:39.156830 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:39.157373 (+ 543us) server_negotiation.cc:299] Negotiation successful
0810 03:50:39.157496 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 03:50:39.158530 3201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:39.145077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:44675 (local address 127.27.153.67:33939)
0810 03:50:39.145624 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:39.145681 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:39.146993 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:39.147578 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:39.147586 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:39.147979 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:39.148495 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:39.148511 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:39.149380 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:39.149383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:39.150354 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:39.150366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:39.150937 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:39.150941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:39.151114 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:39.151835 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:39.151853 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:39.153568 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:39.155404 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:39.155410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:39.155425 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:39.155779 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:39.156167 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:39.156169 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:39.156171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:39.156345 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:39.156851 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:39.156857 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:39.157155 (+ 298us) client_negotiation.cc:769] Sending connection context
0810 03:50:39.157326 (+ 171us) client_negotiation.cc:240] Negotiation successful
0810 03:50:39.157637 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":141,"threads_started":1}
I20260810 03:50:39.159794 3199 heartbeater.cc:347] Connected to a master server at 127.27.153.126:44675
I20260810 03:50:39.160058 3199 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:39.160622 3199 heartbeater.cc:511] Master 127.27.153.126:44675 requested a full tablet report, sending...
I20260810 03:50:39.161815 2378 ts_manager.cc:194] Registered new tserver with Master: d34d1d3504474fa5a3bd794d2bee5da3 (127.27.153.67:41199)
I20260810 03:50:39.162429 2378 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:33939
I20260810 03:50:39.166494 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 03:50:39 dist-test-slave-pww0 krb5kdc[2293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 03:50:39 dist-test-slave-pww0 krb5kdc[2293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.65@KRBTEST.COM
I20260810 03:50:39.195750 3209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:39.185450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47366 (local address 127.27.153.65:38623)
0810 03:50:39.185787 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:39.185793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:39.185821 (+ 28us) server_negotiation.cc:407] Connection header received
0810 03:50:39.185879 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:39.185884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:39.186078 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:39.186205 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:39.186949 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:39.187870 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:39.188788 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:39.189211 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:39.189301 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:39.191933 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:39.191947 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:39.191963 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:39.191993 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:39.193595 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:39.194133 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:39.194139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:39.194141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:39.194257 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:39.194591 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:39.194595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:39.194596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:39.194914 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:39.195120 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:39.195390 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 03:50:39.195597 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":87,"threads_started":1}
Aug 10 03:50:39 dist-test-slave-pww0 krb5kdc[2293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:39.207927 2457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:39.198895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43042 (local address 127.27.153.126:44675)
0810 03:50:39.199054 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:39.199059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:39.199127 (+ 68us) server_negotiation.cc:407] Connection header received
0810 03:50:39.199309 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:39.199313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:39.199360 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:39.199428 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:39.200114 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:39.200636 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:39.201434 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:39.201652 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:39.201750 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:39.204589 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:39.204592 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:39.204594 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:39.204628 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:39.206019 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:39.206509 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:39.206512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:39.206514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:39.206632 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:39.206928 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:39.206932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:39.206933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:39.207190 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:39.207317 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:39.207628 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 03:50:39.207752 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 03:50:39.208540 2378 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43042
I20260810 03:50:39.209268 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 2499
I20260810 03:50:39.222473 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 2735
I20260810 03:50:39.256186 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 2976
I20260810 03:50:39.277254 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 2319
2026-08-10T03:50:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2489 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:50:39 dist-test-slave-pww0 krb5kdc[3218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:50:39 dist-test-slave-pww0 krb5kdc[3218](info): set up 2 sockets
Aug 10 03:50:39 dist-test-slave-pww0 krb5kdc[3218](info): commencing operation
krb5kdc: starting...
W20260810 03:50:40.493072 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:50:40 dist-test-slave-pww0 krb5kdc[3218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:50:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:50:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:50:40.626875 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:45745
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:45745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:40.750389 3235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:40.750715 3235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:40.750782 3235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:40.755096 3235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:50:40.755183 3235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:40.755213 3235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:40.755234 3235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:50:40.755255 3235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:50:40.761015 3235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:45745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:45745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:40.762243 3235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:40.763326 3235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:40.772081 3243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:40.772095 3246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:40.772318 3235 server_base.cc:1034] running on GCE node
W20260810 03:50:40.772094 3249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:40.773006 3235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:40.774232 3235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:40.775413 3235 hybrid_clock.cc:648] HybridClock initialized: now 1786333840775397 us; error 33 us; skew 500 ppm
Aug 10 03:50:40 dist-test-slave-pww0 krb5kdc[3218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333840, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:40.778357 3235 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:50:40.779117 3235 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:40.779191 3235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:40.779393 3235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:40.781993 3235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "070af3e623e44530b082e21ca4e1094b"
format_stamp: "Formatted at 2026-08-10 03:50:40 on dist-test-slave-pww0"
server_key: "bcb0a80dc760566db70300c6af61bcb4"
server_key_iv: "c2ae564808224922e8f2e37afb0d98ea"
server_key_version: "encryptionkey@0"
I20260810 03:50:40.782691 3235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "070af3e623e44530b082e21ca4e1094b"
format_stamp: "Formatted at 2026-08-10 03:50:40 on dist-test-slave-pww0"
server_key: "bcb0a80dc760566db70300c6af61bcb4"
server_key_iv: "c2ae564808224922e8f2e37afb0d98ea"
server_key_version: "encryptionkey@0"
I20260810 03:50:40.787317 3235 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 03:50:40.790623 3261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:40.792272 3235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:40.792440 3235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "070af3e623e44530b082e21ca4e1094b"
format_stamp: "Formatted at 2026-08-10 03:50:40 on dist-test-slave-pww0"
server_key: "bcb0a80dc760566db70300c6af61bcb4"
server_key_iv: "c2ae564808224922e8f2e37afb0d98ea"
server_key_version: "encryptionkey@0"
I20260810 03:50:40.792574 3235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:40.844029 3235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:40.848011 3235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:40.848209 3235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:40.855444 3235 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:45745
I20260810 03:50:40.855868 3349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:45745 every 8 connection(s)
I20260810 03:50:40.856221 3235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:50:40.859596 3351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:40.865866 3351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b: Bootstrap starting.
I20260810 03:50:40.866413 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 3235
I20260810 03:50:40.866557 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:50:40.866855 28261 external_mini_cluster.cc:1428] Setting key 969a8227ed4a7c479d292aec854b969e
I20260810 03:50:40.869215 3351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:40.870028 3351 log.cc:824] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:40.873276 3351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b: No bootstrap required, opened a new log
Aug 10 03:50:40 dist-test-slave-pww0 krb5kdc[3218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:40.877704 3351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070af3e623e44530b082e21ca4e1094b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45745 } }
I20260810 03:50:40.878089 3351 raft_consensus.cc:374] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:40.878167 3351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070af3e623e44530b082e21ca4e1094b, State: Initialized, Role: FOLLOWER
I20260810 03:50:40.878794 3351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070af3e623e44530b082e21ca4e1094b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45745 } }
I20260810 03:50:40.878984 3351 raft_consensus.cc:388] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:50:40.879060 3351 raft_consensus.cc:482] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:50:40.879176 3351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:40.883142 3356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:40.868425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51076 (local address 127.27.153.126:45745)
0810 03:50:40.868938 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:40.868955 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:40.868994 (+ 39us) server_negotiation.cc:407] Connection header received
0810 03:50:40.869776 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:40.869799 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:40.870188 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:40.870556 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:40.871511 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:40.873008 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:40.874020 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:40.874729 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:40.874914 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:40.877594 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:40.877602 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:40.877621 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:40.877660 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:40.879918 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:40.880410 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:40.880415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:40.880420 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:40.880547 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:40.880870 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:40.880873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:40.880875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:40.881351 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:40.881514 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:40.881889 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 03:50:40.882163 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":144,"threads_started":1}
I20260810 03:50:40.885352 3351 raft_consensus.cc:504] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070af3e623e44530b082e21ca4e1094b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45745 } }
I20260810 03:50:40.885874 3351 leader_election.cc:302] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 070af3e623e44530b082e21ca4e1094b; no voters:
I20260810 03:50:40.886310 3351 leader_election.cc:288] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:50:40.886437 3361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:40.886652 3361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b [term 1 LEADER]: Becoming Leader. State: Replica: 070af3e623e44530b082e21ca4e1094b, State: Running, Role: LEADER
I20260810 03:50:40.887092 3361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070af3e623e44530b082e21ca4e1094b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45745 } }
I20260810 03:50:40.887785 3351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:50:40.888428 3365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 070af3e623e44530b082e21ca4e1094b. Latest consensus state: current_term: 1 leader_uuid: "070af3e623e44530b082e21ca4e1094b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070af3e623e44530b082e21ca4e1094b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45745 } } }
I20260810 03:50:40.888566 3365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:40.888896 3366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "070af3e623e44530b082e21ca4e1094b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070af3e623e44530b082e21ca4e1094b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45745 } } }
I20260810 03:50:40.888990 3366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:40.889509 3374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:50:40.892901 3374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:50:40.899480 3374 catalog_manager.cc:1347] Generated new cluster ID: 5c23d3abecc840a88ac53c1d3f1385ae
I20260810 03:50:40.899587 3374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:50:40.921844 3374 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:50:40.923388 3374 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:50:40.932341 3374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b: Generated new TSK 0
I20260810 03:50:40.933511 3374 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.153.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
W20260810 03:50:41.868537 28261 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.27.153.65: real 0.923s user 0.000s sys 0.005s
I20260810 03:50:41.869067 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:45745
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:41.987493 3404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:41.987912 3404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:41.987978 3404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:41.992504 3404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:41.992585 3404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:41.992654 3404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:50:41.998507 3404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:45745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:41.999786 3404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:42.000888 3404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:42.011157 3412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:42.011217 3404 server_base.cc:1034] running on GCE node
W20260810 03:50:42.011162 3411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:42.011157 3414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:42.011931 3404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:42.012646 3404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:42.013823 3404 hybrid_clock.cc:648] HybridClock initialized: now 1786333842013804 us; error 36 us; skew 500 ppm
Aug 10 03:50:42 dist-test-slave-pww0 krb5kdc[3218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333842, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:42.017000 3404 init.cc:377] Logged in from keytab as kudu/127.27.153.65@KRBTEST.COM (short username kudu)
I20260810 03:50:42.017856 3404 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:42.017932 3404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:42.018149 3404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:42.020483 3404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "bf7fcf25cd9147b2b4e842b9f70a545c"
format_stamp: "Formatted at 2026-08-10 03:50:42 on dist-test-slave-pww0"
server_key: "535b0c742663a97a6966e408c929f575"
server_key_iv: "65c13d3339e97017663f9d22c5e41920"
server_key_version: "encryptionkey@0"
I20260810 03:50:42.021090 3404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "bf7fcf25cd9147b2b4e842b9f70a545c"
format_stamp: "Formatted at 2026-08-10 03:50:42 on dist-test-slave-pww0"
server_key: "535b0c742663a97a6966e408c929f575"
server_key_iv: "65c13d3339e97017663f9d22c5e41920"
server_key_version: "encryptionkey@0"
I20260810 03:50:42.025197 3404 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:50:42.028044 3425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:42.029444 3404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:42.029587 3404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "bf7fcf25cd9147b2b4e842b9f70a545c"
format_stamp: "Formatted at 2026-08-10 03:50:42 on dist-test-slave-pww0"
server_key: "535b0c742663a97a6966e408c929f575"
server_key_iv: "65c13d3339e97017663f9d22c5e41920"
server_key_version: "encryptionkey@0"
I20260810 03:50:42.029703 3404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:42.045298 3404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:42.048857 3404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:42.049063 3404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:42.049708 3404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:42.050645 3404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:42.050705 3404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:42.050748 3404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:42.050777 3404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:42.059816 3404 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:39993
I20260810 03:50:42.059850 3623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:39993 every 8 connection(s)
I20260810 03:50:42.060478 3404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:50:42.067363 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 3404
I20260810 03:50:42.067515 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:50:42.067859 28261 external_mini_cluster.cc:1428] Setting key 7971265e0c498350434cce22e303df5f
Aug 10 03:50:42 dist-test-slave-pww0 krb5kdc[3218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333842, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:42.076654 3626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:42.062444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:54389 (local address 127.27.153.126:45745)
0810 03:50:42.062707 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:42.062712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:42.063501 (+ 789us) server_negotiation.cc:407] Connection header received
0810 03:50:42.064919 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:42.064923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:42.064991 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:42.065077 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:42.066807 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.067374 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.068941 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.069177 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.069299 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:42.072045 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:42.072049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:42.072052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:42.072088 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:42.073912 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:42.074575 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:42.074579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:42.074581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:42.074717 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:42.075225 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:42.075229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:42.075231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:42.075544 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:42.075658 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:42.076285 (+ 627us) server_negotiation.cc:299] Negotiation successful
0810 03:50:42.076451 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":86,"threads_started":1}
I20260810 03:50:42.077461 3628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:42.062805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45745 (local address 127.27.153.65:54389)
0810 03:50:42.063299 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:42.063348 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:42.064680 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:42.065298 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:42.065306 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:42.066029 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:42.066602 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:42.066619 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.067561 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.067565 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:42.068820 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:42.068833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.069367 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.069372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:42.069599 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:42.070217 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:42.070235 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:42.071823 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:42.074077 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:42.074083 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:42.074101 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:42.074427 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:42.074853 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:42.074856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:42.074858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:42.075096 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:42.075696 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:42.075703 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:42.076000 (+ 297us) client_negotiation.cc:769] Sending connection context
0810 03:50:42.076177 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 03:50:42.076516 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":109,"threads_started":1}
I20260810 03:50:42.078826 3624 heartbeater.cc:347] Connected to a master server at 127.27.153.126:45745
I20260810 03:50:42.079066 3624 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.153.66@KRBTEST.COM; defaulting to no policy
I20260810 03:50:42.079599 3624 heartbeater.cc:511] Master 127.27.153.126:45745 requested a full tablet report, sending...
I20260810 03:50:42.081372 3290 ts_manager.cc:194] Registered new tserver with Master: bf7fcf25cd9147b2b4e842b9f70a545c (127.27.153.65:39993)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.66@KRBTEST.COM" created.
I20260810 03:50:42.082839 3290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:54389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
I20260810 03:50:42.102414 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:45745
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:42.221873 3634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:42.222175 3634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:42.222239 3634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:42.226439 3634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:42.226526 3634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:42.226616 3634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:50:42.232352 3634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:45745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:42.233590 3634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:42.234741 3634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:42.244762 3642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:42.244922 3634 server_base.cc:1034] running on GCE node
W20260810 03:50:42.244766 3647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:42.244763 3643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:42.245570 3634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:42.246309 3634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:42.247532 3634 hybrid_clock.cc:648] HybridClock initialized: now 1786333842247504 us; error 39 us; skew 500 ppm
Aug 10 03:50:42 dist-test-slave-pww0 krb5kdc[3218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333842, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:42.250761 3634 init.cc:377] Logged in from keytab as kudu/127.27.153.66@KRBTEST.COM (short username kudu)
I20260810 03:50:42.251714 3634 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:42.251802 3634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:42.252035 3634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:42.254280 3634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "f70bfa4750f742e0a127097678c0915d"
format_stamp: "Formatted at 2026-08-10 03:50:42 on dist-test-slave-pww0"
server_key: "9b75fac24f3c674ac5f16632be81cf2f"
server_key_iv: "3a33cce0f51ddbd3e2c57ea489d6ac25"
server_key_version: "encryptionkey@0"
I20260810 03:50:42.254885 3634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "f70bfa4750f742e0a127097678c0915d"
format_stamp: "Formatted at 2026-08-10 03:50:42 on dist-test-slave-pww0"
server_key: "9b75fac24f3c674ac5f16632be81cf2f"
server_key_iv: "3a33cce0f51ddbd3e2c57ea489d6ac25"
server_key_version: "encryptionkey@0"
I20260810 03:50:42.259029 3634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:50:42.261798 3655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:42.263141 3634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:50:42.263274 3634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "f70bfa4750f742e0a127097678c0915d"
format_stamp: "Formatted at 2026-08-10 03:50:42 on dist-test-slave-pww0"
server_key: "9b75fac24f3c674ac5f16632be81cf2f"
server_key_iv: "3a33cce0f51ddbd3e2c57ea489d6ac25"
server_key_version: "encryptionkey@0"
I20260810 03:50:42.263408 3634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:42.280857 3634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:42.284245 3634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:42.284451 3634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:42.285099 3634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:42.286038 3634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:42.286095 3634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:42.286160 3634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:42.286188 3634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:42.295797 3634 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:33109
I20260810 03:50:42.295825 3843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:33109 every 8 connection(s)
I20260810 03:50:42.296494 3634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:50:42.300519 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 3634
I20260810 03:50:42.300673 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:50:42.300987 28261 external_mini_cluster.cc:1428] Setting key b15fd0e865164d60efdb4c1894abe505
Aug 10 03:50:42 dist-test-slave-pww0 krb5kdc[3218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333842, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.153.67@KRBTEST.COM; defaulting to no policy
I20260810 03:50:42.313453 3626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:42.298497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:52803 (local address 127.27.153.126:45745)
0810 03:50:42.298660 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:42.298665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:42.299791 (+ 1126us) server_negotiation.cc:407] Connection header received
0810 03:50:42.301553 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:42.301558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:42.301628 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:42.301809 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:42.303261 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.303893 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.305432 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.305787 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.305912 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:42.309034 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:42.309039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:42.309042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:42.309082 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:42.310825 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:42.311536 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:42.311540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:42.311543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:42.311650 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:42.312138 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:42.312143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:42.312144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:42.312419 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:42.312532 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:42.313170 (+ 638us) server_negotiation.cc:299] Negotiation successful
0810 03:50:42.313305 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:50:42.314342 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:42.298974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45745 (local address 127.27.153.66:52803)
0810 03:50:42.299599 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:42.299669 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:42.301248 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:42.301954 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:42.301964 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:42.302375 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:42.303077 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:42.303102 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.304030 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.304033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:42.305284 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:42.305295 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.305992 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.305995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:42.306201 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:42.306868 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:42.306887 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:42.308789 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:42.310971 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:42.310977 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:42.310994 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:42.311395 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:42.311762 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:42.311766 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:42.311769 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:42.312002 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:42.312549 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:42.312556 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:42.312861 (+ 305us) client_negotiation.cc:769] Sending connection context
0810 03:50:42.313047 (+ 186us) client_negotiation.cc:240] Negotiation successful
0810 03:50:42.313375 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":140,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.67@KRBTEST.COM" created.
I20260810 03:50:42.315758 3844 heartbeater.cc:347] Connected to a master server at 127.27.153.126:45745
I20260810 03:50:42.316021 3844 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:42.316545 3844 heartbeater.cc:511] Master 127.27.153.126:45745 requested a full tablet report, sending...
I20260810 03:50:42.317749 3290 ts_manager.cc:194] Registered new tserver with Master: f70bfa4750f742e0a127097678c0915d (127.27.153.66:33109)
I20260810 03:50:42.318364 3290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:52803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
I20260810 03:50:42.332314 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:45745
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:42.450984 3852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:42.451287 3852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:42.451334 3852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:42.455543 3852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:42.455631 3852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:42.455695 3852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:50:42.461337 3852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:45745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:42.462505 3852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:42.463599 3852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:50:42.474100 3852 server_base.cc:1034] running on GCE node
W20260810 03:50:42.474159 3860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:42.474157 3859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:42.474318 3863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:42.474898 3852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:42.475694 3852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:42.476864 3852 hybrid_clock.cc:648] HybridClock initialized: now 1786333842476844 us; error 39 us; skew 500 ppm
Aug 10 03:50:42 dist-test-slave-pww0 krb5kdc[3218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333842, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:42.479924 3852 init.cc:377] Logged in from keytab as kudu/127.27.153.67@KRBTEST.COM (short username kudu)
I20260810 03:50:42.480731 3852 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:42.480818 3852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:42.481038 3852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:42.483213 3852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "de679ca547314b3e9547b305c4596ba8"
format_stamp: "Formatted at 2026-08-10 03:50:42 on dist-test-slave-pww0"
server_key: "301304b7962f73abc9c84d8cd6061316"
server_key_iv: "e69f501170f12a719fc873bc658d53a7"
server_key_version: "encryptionkey@0"
I20260810 03:50:42.483850 3852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "de679ca547314b3e9547b305c4596ba8"
format_stamp: "Formatted at 2026-08-10 03:50:42 on dist-test-slave-pww0"
server_key: "301304b7962f73abc9c84d8cd6061316"
server_key_iv: "e69f501170f12a719fc873bc658d53a7"
server_key_version: "encryptionkey@0"
I20260810 03:50:42.487977 3852 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:50:42.490799 3875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:42.492147 3852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:42.492271 3852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "de679ca547314b3e9547b305c4596ba8"
format_stamp: "Formatted at 2026-08-10 03:50:42 on dist-test-slave-pww0"
server_key: "301304b7962f73abc9c84d8cd6061316"
server_key_iv: "e69f501170f12a719fc873bc658d53a7"
server_key_version: "encryptionkey@0"
I20260810 03:50:42.492384 3852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:42.515434 3852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:42.519011 3852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:42.519201 3852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:42.519886 3852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:42.520807 3852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:42.520855 3852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:42.520911 3852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:42.520941 3852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:42.530081 3852 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:40783
I20260810 03:50:42.530125 4086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:40783 every 8 connection(s)
I20260810 03:50:42.530764 3852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:50:42.540869 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 3852
I20260810 03:50:42.541018 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:50:42.541386 28261 external_mini_cluster.cc:1428] Setting key 1a392e9dbc055981e3e267a6fc2c393c
Aug 10 03:50:42 dist-test-slave-pww0 krb5kdc[3218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333842, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:42.546536 3626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:42.532722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:57855 (local address 127.27.153.126:45745)
0810 03:50:42.532899 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:42.532903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:42.533956 (+ 1053us) server_negotiation.cc:407] Connection header received
0810 03:50:42.535425 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:42.535433 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:42.535531 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:42.535656 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:42.537081 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.537636 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.538990 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.539230 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.539328 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:42.542550 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:42.542554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:42.542557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:42.542593 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:42.544318 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:42.544976 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:42.544979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:42.544981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:42.545076 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:42.545433 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:42.545436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:42.545437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:42.545648 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:42.545737 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:42.546288 (+ 551us) server_negotiation.cc:299] Negotiation successful
0810 03:50:42.546408 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 03:50:42.547420 4089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:42.533186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45745 (local address 127.27.153.67:57855)
0810 03:50:42.533786 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:42.533842 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:42.535165 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:42.535875 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:42.535883 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:42.536295 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:42.536864 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:42.536879 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.537822 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.537826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:42.538830 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:42.538842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.539422 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.539425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:42.539684 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:42.540356 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:42.540375 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:42.542280 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:42.544469 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:42.544476 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:42.544493 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:42.544872 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:42.545164 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:42.545167 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:42.545168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:42.545342 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:42.545758 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:42.545769 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:42.546057 (+ 288us) client_negotiation.cc:769] Sending connection context
0810 03:50:42.546230 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 03:50:42.546527 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":134,"threads_started":1}
I20260810 03:50:42.548698 4087 heartbeater.cc:347] Connected to a master server at 127.27.153.126:45745
I20260810 03:50:42.548977 4087 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:42.549499 4087 heartbeater.cc:511] Master 127.27.153.126:45745 requested a full tablet report, sending...
I20260810 03:50:42.550644 3290 ts_manager.cc:194] Registered new tserver with Master: de679ca547314b3e9547b305c4596ba8 (127.27.153.67:40783)
I20260810 03:50:42.551260 3290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:57855
I20260810 03:50:42.556275 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:50:42.565932 3626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:42.558243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37404 (local address 127.27.153.126:45745)
0810 03:50:42.558389 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:42.558395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:42.558560 (+ 165us) server_negotiation.cc:407] Connection header received
0810 03:50:42.558698 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:42.558701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:42.558750 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:42.558816 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:42.559588 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.560115 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.560957 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.561199 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.561291 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:42.562634 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:42.562638 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:42.562641 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:42.562678 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:42.564147 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:42.564606 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:42.564610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:42.564612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:42.564706 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:42.564979 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:42.564983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:42.564985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:42.565257 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:42.565389 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:42.565637 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 03:50:42.565767 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:50:42.569253 3290 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37404:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:50:42.572417 3290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:50:42.587946 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:42.582679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46144 (local address 127.27.153.67:40783)
0810 03:50:42.583119 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:42.583135 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:42.583493 (+ 358us) server_negotiation.cc:407] Connection header received
0810 03:50:42.583700 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:42.583705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:42.583895 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:42.584043 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:42.584282 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.585269 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.586548 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.587135 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.587251 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:42.587422 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:42.587642 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 03:50:42.587769 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":152,"threads_started":1}
I20260810 03:50:42.588101 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:42.582038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56068 (local address 127.27.153.65:39993)
0810 03:50:42.582406 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:42.582412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:42.582979 (+ 567us) server_negotiation.cc:407] Connection header received
0810 03:50:42.583067 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:42.583073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:42.583275 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:42.583412 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:42.583811 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.584701 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.586517 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.587383 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.587534 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:42.587689 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:42.587829 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 03:50:42.587932 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":92,"threads_started":1}
I20260810 03:50:42.588479 4107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:42.581611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:33109 (local address 127.0.0.1:34704)
0810 03:50:42.582528 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:42.582598 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:42.582752 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:42.583426 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:42.583432 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:42.583501 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:42.583641 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:42.583648 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.585061 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.585064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:42.586371 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:42.586383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.587543 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.587547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:42.588115 (+ 568us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:42.588141 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 03:50:42.588211 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 03:50:42.588297 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"mutex_wait_us":504,"thread_start_us":141,"threads_started":1}
I20260810 03:50:42.588546 4108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:42.582448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:40783 (local address 127.0.0.1:46144)
0810 03:50:42.583367 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:42.583381 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:42.583565 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:42.584039 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:42.584041 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:42.584062 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:42.584162 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:42.584168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.585416 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.585420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:42.586437 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:42.586449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.587265 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.587268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:42.587391 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:42.587433 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 03:50:42.588213 (+ 780us) client_negotiation.cc:240] Negotiation successful
0810 03:50:42.588297 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":5,"thread_start_us":112,"threads_started":1}
I20260810 03:50:42.588778 4109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:42.581772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34704 (local address 127.27.153.66:33109)
0810 03:50:42.582258 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:42.582272 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:42.582924 (+ 652us) server_negotiation.cc:407] Connection header received
0810 03:50:42.583014 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:42.583019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:42.583246 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:42.583415 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:42.583763 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.584924 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.586603 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.587380 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.587673 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:42.587813 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:42.588482 (+ 669us) server_negotiation.cc:299] Negotiation successful
0810 03:50:42.588626 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":130,"threads_started":1}
I20260810 03:50:42.588831 4111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:42.581770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:39993 (local address 127.0.0.1:56068)
0810 03:50:42.582773 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:42.582786 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:42.582878 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:42.583590 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:42.583592 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:42.583607 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:42.583692 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:42.583697 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.584812 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.584814 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:42.586371 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:42.586383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.587554 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.587557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:42.587691 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:42.587715 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:50:42.587785 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 03:50:42.587925 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"mutex_wait_us":149,"thread_start_us":77,"threads_started":1}
I20260810 03:50:42.591703 3735 tablet_service.cc:1467] Processing CreateTablet for tablet 9de54dc57ada4b08944dbb6e980945b2 (DEFAULT_TABLE table=test-table [id=8240eb444c214af5a6efd313ec246822]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:42.592941 3735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9de54dc57ada4b08944dbb6e980945b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:42.593598 3504 tablet_service.cc:1467] Processing CreateTablet for tablet 9de54dc57ada4b08944dbb6e980945b2 (DEFAULT_TABLE table=test-table [id=8240eb444c214af5a6efd313ec246822]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:42.594815 3504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9de54dc57ada4b08944dbb6e980945b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:42.594527 3967 tablet_service.cc:1467] Processing CreateTablet for tablet 9de54dc57ada4b08944dbb6e980945b2 (DEFAULT_TABLE table=test-table [id=8240eb444c214af5a6efd313ec246822]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:42.595691 3967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9de54dc57ada4b08944dbb6e980945b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:42.601301 4114 tablet_bootstrap.cc:492] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d: Bootstrap starting.
I20260810 03:50:42.602660 4115 tablet_bootstrap.cc:492] T 9de54dc57ada4b08944dbb6e980945b2 P bf7fcf25cd9147b2b4e842b9f70a545c: Bootstrap starting.
I20260810 03:50:42.603551 4117 tablet_bootstrap.cc:492] T 9de54dc57ada4b08944dbb6e980945b2 P de679ca547314b3e9547b305c4596ba8: Bootstrap starting.
I20260810 03:50:42.604854 4114 tablet_bootstrap.cc:654] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:42.605167 4115 tablet_bootstrap.cc:654] T 9de54dc57ada4b08944dbb6e980945b2 P bf7fcf25cd9147b2b4e842b9f70a545c: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:42.605743 4117 tablet_bootstrap.cc:654] T 9de54dc57ada4b08944dbb6e980945b2 P de679ca547314b3e9547b305c4596ba8: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:42.606110 4115 log.cc:824] T 9de54dc57ada4b08944dbb6e980945b2 P bf7fcf25cd9147b2b4e842b9f70a545c: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:42.606243 4114 log.cc:824] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:42.607074 4117 log.cc:824] T 9de54dc57ada4b08944dbb6e980945b2 P de679ca547314b3e9547b305c4596ba8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:42.608557 4115 tablet_bootstrap.cc:492] T 9de54dc57ada4b08944dbb6e980945b2 P bf7fcf25cd9147b2b4e842b9f70a545c: No bootstrap required, opened a new log
I20260810 03:50:42.608557 4114 tablet_bootstrap.cc:492] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d: No bootstrap required, opened a new log
I20260810 03:50:42.608793 4114 ts_tablet_manager.cc:1397] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 03:50:42.608793 4115 ts_tablet_manager.cc:1397] T 9de54dc57ada4b08944dbb6e980945b2 P bf7fcf25cd9147b2b4e842b9f70a545c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:50:42.609171 4117 tablet_bootstrap.cc:492] T 9de54dc57ada4b08944dbb6e980945b2 P de679ca547314b3e9547b305c4596ba8: No bootstrap required, opened a new log
I20260810 03:50:42.609382 4117 ts_tablet_manager.cc:1397] T 9de54dc57ada4b08944dbb6e980945b2 P de679ca547314b3e9547b305c4596ba8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:50:42.612623 4117 raft_consensus.cc:348] T 9de54dc57ada4b08944dbb6e980945b2 P de679ca547314b3e9547b305c4596ba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70bfa4750f742e0a127097678c0915d" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 33109 } } peers { permanent_uuid: "bf7fcf25cd9147b2b4e842b9f70a545c" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39993 } } peers { permanent_uuid: "de679ca547314b3e9547b305c4596ba8" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 40783 } }
I20260810 03:50:42.612957 4117 raft_consensus.cc:374] T 9de54dc57ada4b08944dbb6e980945b2 P de679ca547314b3e9547b305c4596ba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:42.613014 4117 raft_consensus.cc:729] T 9de54dc57ada4b08944dbb6e980945b2 P de679ca547314b3e9547b305c4596ba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de679ca547314b3e9547b305c4596ba8, State: Initialized, Role: FOLLOWER
I20260810 03:50:42.613376 4115 raft_consensus.cc:348] T 9de54dc57ada4b08944dbb6e980945b2 P bf7fcf25cd9147b2b4e842b9f70a545c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70bfa4750f742e0a127097678c0915d" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 33109 } } peers { permanent_uuid: "bf7fcf25cd9147b2b4e842b9f70a545c" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39993 } } peers { permanent_uuid: "de679ca547314b3e9547b305c4596ba8" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 40783 } }
I20260810 03:50:42.613358 4114 raft_consensus.cc:348] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70bfa4750f742e0a127097678c0915d" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 33109 } } peers { permanent_uuid: "bf7fcf25cd9147b2b4e842b9f70a545c" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39993 } } peers { permanent_uuid: "de679ca547314b3e9547b305c4596ba8" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 40783 } }
I20260810 03:50:42.613524 4117 consensus_queue.cc:260] T 9de54dc57ada4b08944dbb6e980945b2 P de679ca547314b3e9547b305c4596ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70bfa4750f742e0a127097678c0915d" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 33109 } } peers { permanent_uuid: "bf7fcf25cd9147b2b4e842b9f70a545c" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39993 } } peers { permanent_uuid: "de679ca547314b3e9547b305c4596ba8" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 40783 } }
I20260810 03:50:42.613647 4115 raft_consensus.cc:374] T 9de54dc57ada4b08944dbb6e980945b2 P bf7fcf25cd9147b2b4e842b9f70a545c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:42.613660 4114 raft_consensus.cc:374] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:42.613718 4115 raft_consensus.cc:729] T 9de54dc57ada4b08944dbb6e980945b2 P bf7fcf25cd9147b2b4e842b9f70a545c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf7fcf25cd9147b2b4e842b9f70a545c, State: Initialized, Role: FOLLOWER
I20260810 03:50:42.613718 4114 raft_consensus.cc:729] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f70bfa4750f742e0a127097678c0915d, State: Initialized, Role: FOLLOWER
I20260810 03:50:42.614205 4087 heartbeater.cc:503] Master 127.27.153.126:45745 was elected leader, sending a full tablet report...
I20260810 03:50:42.614187 4115 consensus_queue.cc:260] T 9de54dc57ada4b08944dbb6e980945b2 P bf7fcf25cd9147b2b4e842b9f70a545c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70bfa4750f742e0a127097678c0915d" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 33109 } } peers { permanent_uuid: "bf7fcf25cd9147b2b4e842b9f70a545c" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39993 } } peers { permanent_uuid: "de679ca547314b3e9547b305c4596ba8" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 40783 } }
I20260810 03:50:42.614187 4114 consensus_queue.cc:260] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70bfa4750f742e0a127097678c0915d" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 33109 } } peers { permanent_uuid: "bf7fcf25cd9147b2b4e842b9f70a545c" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39993 } } peers { permanent_uuid: "de679ca547314b3e9547b305c4596ba8" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 40783 } }
I20260810 03:50:42.615170 3624 heartbeater.cc:503] Master 127.27.153.126:45745 was elected leader, sending a full tablet report...
I20260810 03:50:42.615271 4115 ts_tablet_manager.cc:1428] T 9de54dc57ada4b08944dbb6e980945b2 P bf7fcf25cd9147b2b4e842b9f70a545c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 03:50:42.615335 3844 heartbeater.cc:503] Master 127.27.153.126:45745 was elected leader, sending a full tablet report...
I20260810 03:50:42.616504 4114 ts_tablet_manager.cc:1428] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260810 03:50:42.617748 4117 ts_tablet_manager.cc:1428] T 9de54dc57ada4b08944dbb6e980945b2 P de679ca547314b3e9547b305c4596ba8: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260810 03:50:42.745378 4124 raft_consensus.cc:482] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:50:42.745609 4124 raft_consensus.cc:504] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70bfa4750f742e0a127097678c0915d" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 33109 } } peers { permanent_uuid: "bf7fcf25cd9147b2b4e842b9f70a545c" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39993 } } peers { permanent_uuid: "de679ca547314b3e9547b305c4596ba8" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 40783 } }
I20260810 03:50:42.746846 4124 leader_election.cc:288] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf7fcf25cd9147b2b4e842b9f70a545c (127.27.153.65:39993), de679ca547314b3e9547b305c4596ba8 (127.27.153.67:40783)
I20260810 03:50:42.751096 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:42.747150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:39993 (local address 127.27.153.66:52421)
0810 03:50:42.747287 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:42.747303 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:42.747417 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:42.747790 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:42.747793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:42.747820 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:42.747927 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:42.747933 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.748855 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.748858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:42.749917 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:42.749928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.750758 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.750760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:42.750849 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:42.750871 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 03:50:42.750926 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 03:50:42.750985 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 03:50:42.751207 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:42.747239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:52421 (local address 127.27.153.65:39993)
0810 03:50:42.747365 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:42.747368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:42.747386 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:50:42.747622 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:42.747626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:42.747685 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:42.747763 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:42.748074 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.748740 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.750085 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.750619 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.750783 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:42.750849 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:42.750963 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 03:50:42.751066 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 03:50:42.751845 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:42.747432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:40783 (local address 127.27.153.66:57457)
0810 03:50:42.748203 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:42.748215 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:42.748303 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:42.748502 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:42.748504 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:42.748515 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:42.748606 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:42.748610 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.749319 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.749323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:42.750334 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:42.750343 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.751399 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.751403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:42.751536 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:42.751577 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 03:50:42.751663 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 03:50:42.751728 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":171,"threads_started":1}
I20260810 03:50:42.751899 4112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:42.747582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:57457 (local address 127.27.153.67:40783)
0810 03:50:42.747730 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:42.747735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:42.748301 (+ 566us) server_negotiation.cc:407] Connection header received
0810 03:50:42.748364 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:42.748368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:42.748416 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:42.748493 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:42.748689 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.749204 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.750483 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.751120 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.751250 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:42.751306 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:42.751700 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 03:50:42.751786 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 03:50:42.751976 3536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de54dc57ada4b08944dbb6e980945b2" candidate_uuid: "f70bfa4750f742e0a127097678c0915d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf7fcf25cd9147b2b4e842b9f70a545c" is_pre_election: true
I20260810 03:50:42.752440 3536 raft_consensus.cc:2454] T 9de54dc57ada4b08944dbb6e980945b2 P bf7fcf25cd9147b2b4e842b9f70a545c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f70bfa4750f742e0a127097678c0915d in term 0.
I20260810 03:50:42.752575 3994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de54dc57ada4b08944dbb6e980945b2" candidate_uuid: "f70bfa4750f742e0a127097678c0915d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de679ca547314b3e9547b305c4596ba8" is_pre_election: true
I20260810 03:50:42.752851 3994 raft_consensus.cc:2454] T 9de54dc57ada4b08944dbb6e980945b2 P de679ca547314b3e9547b305c4596ba8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f70bfa4750f742e0a127097678c0915d in term 0.
I20260810 03:50:42.753000 3659 leader_election.cc:302] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf7fcf25cd9147b2b4e842b9f70a545c, f70bfa4750f742e0a127097678c0915d; no voters:
I20260810 03:50:42.753294 4124 raft_consensus.cc:2781] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:50:42.753374 4124 raft_consensus.cc:482] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:50:42.753405 4124 raft_consensus.cc:3037] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:42.754503 4124 raft_consensus.cc:504] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70bfa4750f742e0a127097678c0915d" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 33109 } } peers { permanent_uuid: "bf7fcf25cd9147b2b4e842b9f70a545c" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39993 } } peers { permanent_uuid: "de679ca547314b3e9547b305c4596ba8" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 40783 } }
I20260810 03:50:42.754854 4124 leader_election.cc:288] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d [CANDIDATE]: Term 1 election: Requested vote from peers bf7fcf25cd9147b2b4e842b9f70a545c (127.27.153.65:39993), de679ca547314b3e9547b305c4596ba8 (127.27.153.67:40783)
I20260810 03:50:42.755326 3536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de54dc57ada4b08944dbb6e980945b2" candidate_uuid: "f70bfa4750f742e0a127097678c0915d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf7fcf25cd9147b2b4e842b9f70a545c"
I20260810 03:50:42.755396 3994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de54dc57ada4b08944dbb6e980945b2" candidate_uuid: "f70bfa4750f742e0a127097678c0915d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de679ca547314b3e9547b305c4596ba8"
I20260810 03:50:42.755446 3536 raft_consensus.cc:3037] T 9de54dc57ada4b08944dbb6e980945b2 P bf7fcf25cd9147b2b4e842b9f70a545c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:42.755576 3994 raft_consensus.cc:3037] T 9de54dc57ada4b08944dbb6e980945b2 P de679ca547314b3e9547b305c4596ba8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:42.756698 3536 raft_consensus.cc:2454] T 9de54dc57ada4b08944dbb6e980945b2 P bf7fcf25cd9147b2b4e842b9f70a545c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f70bfa4750f742e0a127097678c0915d in term 1.
I20260810 03:50:42.756711 3994 raft_consensus.cc:2454] T 9de54dc57ada4b08944dbb6e980945b2 P de679ca547314b3e9547b305c4596ba8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f70bfa4750f742e0a127097678c0915d in term 1.
I20260810 03:50:42.757032 3659 leader_election.cc:302] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de679ca547314b3e9547b305c4596ba8, f70bfa4750f742e0a127097678c0915d; no voters:
I20260810 03:50:42.757257 4124 raft_consensus.cc:2781] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:42.757496 4124 raft_consensus.cc:686] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d [term 1 LEADER]: Becoming Leader. State: Replica: f70bfa4750f742e0a127097678c0915d, State: Running, Role: LEADER
I20260810 03:50:42.757861 4124 consensus_queue.cc:237] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70bfa4750f742e0a127097678c0915d" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 33109 } } peers { permanent_uuid: "bf7fcf25cd9147b2b4e842b9f70a545c" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39993 } } peers { permanent_uuid: "de679ca547314b3e9547b305c4596ba8" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 40783 } }
I20260810 03:50:42.761221 3290 catalog_manager.cc:5495] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d reported cstate change: term changed from 0 to 1, leader changed from <none> to f70bfa4750f742e0a127097678c0915d (127.27.153.66). New cstate: current_term: 1 leader_uuid: "f70bfa4750f742e0a127097678c0915d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f70bfa4750f742e0a127097678c0915d" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 33109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf7fcf25cd9147b2b4e842b9f70a545c" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de679ca547314b3e9547b305c4596ba8" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 40783 } health_report { overall_health: UNKNOWN } } }
I20260810 03:50:42.780572 4109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:42.776991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34718 (local address 127.27.153.66:33109)
0810 03:50:42.777116 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:42.777121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:42.777137 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:50:42.777222 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:42.777224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:42.777282 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:42.777378 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:50:42.777653 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.778160 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.779025 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:42.779246 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:42.779339 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:42.779716 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:50:42.780168 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:50:42.780283 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:42.780362 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 03:50:42.780427 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 03:50:42.781783 4088 tablet.cc:2369] T 9de54dc57ada4b08944dbb6e980945b2 P de679ca547314b3e9547b305c4596ba8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:50:42.786904 3536 raft_consensus.cc:1261] T 9de54dc57ada4b08944dbb6e980945b2 P bf7fcf25cd9147b2b4e842b9f70a545c [term 1 FOLLOWER]: Refusing update from remote peer f70bfa4750f742e0a127097678c0915d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:42.786904 3994 raft_consensus.cc:1261] T 9de54dc57ada4b08944dbb6e980945b2 P de679ca547314b3e9547b305c4596ba8 [term 1 FOLLOWER]: Refusing update from remote peer f70bfa4750f742e0a127097678c0915d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:42.787698 4124 consensus_queue.cc:1035] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d [LEADER]: Connected to new peer: Peer: permanent_uuid: "de679ca547314b3e9547b305c4596ba8" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 40783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:42.787926 4128 consensus_queue.cc:1035] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf7fcf25cd9147b2b4e842b9f70a545c" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:42.795521 4134 mvcc.cc:205] Tried to move back new op lower bound from 7316823420046700544 to 7316823419939033088. Current Snapshot: MvccSnapshot[applied={T|T < 7316823420046700544}]
I20260810 03:50:42.806470 3290 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:42.806723 3290 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37404:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:42.809623 3290 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 070af3e623e44530b082e21ca4e1094b: Sending DeleteTablet for 3 replicas of tablet 9de54dc57ada4b08944dbb6e980945b2
I20260810 03:50:42.810702 3504 tablet_service.cc:1514] Processing DeleteTablet for tablet 9de54dc57ada4b08944dbb6e980945b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:42 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:56068
I20260810 03:50:42.810739 3967 tablet_service.cc:1514] Processing DeleteTablet for tablet 9de54dc57ada4b08944dbb6e980945b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:42 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:46144
I20260810 03:50:42.811308 4145 tablet_replica.cc:330] T 9de54dc57ada4b08944dbb6e980945b2 P de679ca547314b3e9547b305c4596ba8: stopping tablet replica
I20260810 03:50:42.811309 4146 tablet_replica.cc:330] T 9de54dc57ada4b08944dbb6e980945b2 P bf7fcf25cd9147b2b4e842b9f70a545c: stopping tablet replica
I20260810 03:50:42.811322 3735 tablet_service.cc:1514] Processing DeleteTablet for tablet 9de54dc57ada4b08944dbb6e980945b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:42 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:34704
I20260810 03:50:42.811717 4145 raft_consensus.cc:2229] T 9de54dc57ada4b08944dbb6e980945b2 P de679ca547314b3e9547b305c4596ba8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:42.812095 4145 raft_consensus.cc:2258] T 9de54dc57ada4b08944dbb6e980945b2 P de679ca547314b3e9547b305c4596ba8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:42.812220 4146 raft_consensus.cc:2229] T 9de54dc57ada4b08944dbb6e980945b2 P bf7fcf25cd9147b2b4e842b9f70a545c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:42.812608 4146 raft_consensus.cc:2258] T 9de54dc57ada4b08944dbb6e980945b2 P bf7fcf25cd9147b2b4e842b9f70a545c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:42.812675 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 3404
I20260810 03:50:42.814069 4145 ts_tablet_manager.cc:1905] T 9de54dc57ada4b08944dbb6e980945b2 P de679ca547314b3e9547b305c4596ba8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:42.816586 4145 ts_tablet_manager.cc:1918] T 9de54dc57ada4b08944dbb6e980945b2 P de679ca547314b3e9547b305c4596ba8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:50:42.816707 4145 log.cc:1196] T 9de54dc57ada4b08944dbb6e980945b2 P de679ca547314b3e9547b305c4596ba8: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/9de54dc57ada4b08944dbb6e980945b2
I20260810 03:50:42.816820 4147 tablet_replica.cc:330] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d: stopping tablet replica
I20260810 03:50:42.817051 4145 ts_tablet_manager.cc:1939] T 9de54dc57ada4b08944dbb6e980945b2 P de679ca547314b3e9547b305c4596ba8: Deleting consensus metadata
I20260810 03:50:42.817327 4147 raft_consensus.cc:2229] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:50:42.817826 4147 raft_consensus.cc:2258] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:42.819770 4147 ts_tablet_manager.cc:1905] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:42.822301 4147 ts_tablet_manager.cc:1918] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:50:42.822465 4147 log.cc:1196] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/9de54dc57ada4b08944dbb6e980945b2
I20260810 03:50:42.822757 4147 ts_tablet_manager.cc:1939] T 9de54dc57ada4b08944dbb6e980945b2 P f70bfa4750f742e0a127097678c0915d: Deleting consensus metadata
I20260810 03:50:42.823539 3267 catalog_manager.cc:4841] TS de679ca547314b3e9547b305c4596ba8 (127.27.153.67:40783): tablet 9de54dc57ada4b08944dbb6e980945b2 (table test-table [id=8240eb444c214af5a6efd313ec246822]) successfully deleted
I20260810 03:50:42.824127 3266 catalog_manager.cc:4841] TS f70bfa4750f742e0a127097678c0915d (127.27.153.66:33109): tablet 9de54dc57ada4b08944dbb6e980945b2 (table test-table [id=8240eb444c214af5a6efd313ec246822]) successfully deleted
W20260810 03:50:42.832477 3267 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.153.65:39993 (error 108)
I20260810 03:50:42.833407 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 3634
I20260810 03:50:42.833684 4107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:42.833350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:39993 (local address 127.0.0.1:56082)
0810 03:50:42.833508 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:42.833588 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:39993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 03:50:42.842626 3267 catalog_manager.cc:4584] TS bf7fcf25cd9147b2b4e842b9f70a545c (127.27.153.65:39993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9de54dc57ada4b08944dbb6e980945b2: Network error: Client connection negotiation failed: client connection to 127.27.153.65:39993: connect: Connection refused (error 111)
I20260810 03:50:42.848455 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 3852
I20260810 03:50:42.865818 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 3235
2026-08-10T03:50:42Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3588 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T03:50:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:50:42Z Disabled control of system clock
I20260810 03:50:42.907887 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:42319
--webserver_interface=127.27.153.126
--webserver_port=0
--builtin_ntp_servers=127.27.153.84:40803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:42319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 03:50:43.027987 4155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:43.028283 4155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:43.028322 4155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:43.032531 4155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:50:43.032611 4155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:43.032626 4155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:43.032642 4155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:50:43.032657 4155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:50:43.037829 4155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:42319
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:42319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:43.039021 4155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:43.040582 4155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:43.050038 4166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:43.050149 4173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:43.050119 4155 server_base.cc:1034] running on GCE node
W20260810 03:50:43.050364 4167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:43.051777 4155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:43.052959 4155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:43.054143 4155 hybrid_clock.cc:648] HybridClock initialized: now 1786333843054116 us; error 53 us; skew 500 ppm
I20260810 03:50:43.056383 4155 webserver.cc:460] Webserver started at http://127.27.153.126:37455/ using document root <none> and password file <none>
I20260810 03:50:43.056985 4155 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:43.057058 4155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:43.057247 4155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:43.059538 4155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "cab942440f41439e821fe3dcb8457782"
format_stamp: "Formatted at 2026-08-10 03:50:43 on dist-test-slave-pww0"
server_key: "254d769f1476d47e08011b48a9b737a8"
server_key_iv: "dd0cb42db429fd265fb3760ddcedd0e9"
server_key_version: "encryptionkey@0"
I20260810 03:50:43.060173 4155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "cab942440f41439e821fe3dcb8457782"
format_stamp: "Formatted at 2026-08-10 03:50:43 on dist-test-slave-pww0"
server_key: "254d769f1476d47e08011b48a9b737a8"
server_key_iv: "dd0cb42db429fd265fb3760ddcedd0e9"
server_key_version: "encryptionkey@0"
I20260810 03:50:43.064625 4155 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:50:43.067363 4183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:43.068773 4155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 03:50:43.068925 4155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "cab942440f41439e821fe3dcb8457782"
format_stamp: "Formatted at 2026-08-10 03:50:43 on dist-test-slave-pww0"
server_key: "254d769f1476d47e08011b48a9b737a8"
server_key_iv: "dd0cb42db429fd265fb3760ddcedd0e9"
server_key_version: "encryptionkey@0"
I20260810 03:50:43.069051 4155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:43.074201 4155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:43.075078 4155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:43.075294 4155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:43.082661 4155 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:42319
I20260810 03:50:43.082674 4277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:42319 every 8 connection(s)
I20260810 03:50:43.083787 4155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:50:43.084811 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 4155
I20260810 03:50:43.084918 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:50:43.085237 28261 external_mini_cluster.cc:1428] Setting key 0f675cb53e5cfe54222b3162839d1d82
I20260810 03:50:43.087639 4278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:43.092373 4282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:43.086972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46178 (local address 127.27.153.126:42319)
0810 03:50:43.088406 (+ 1434us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:43.088423 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:43.088464 (+ 41us) server_negotiation.cc:407] Connection header received
0810 03:50:43.089156 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:43.089177 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:43.089660 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:43.090027 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:43.090908 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:43.090913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:43.090926 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:50:43.090938 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:43.090997 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:50:43.091013 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:43.091154 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:43.091265 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 03:50:43.091567 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1005,"thread_start_us":743,"threads_started":1}
I20260810 03:50:43.094426 4278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782: Bootstrap starting.
I20260810 03:50:43.097112 4278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:43.098114 4278 log.cc:824] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:43.100561 4278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782: No bootstrap required, opened a new log
I20260810 03:50:43.104786 4278 raft_consensus.cc:348] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab942440f41439e821fe3dcb8457782" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42319 } }
I20260810 03:50:43.105083 4278 raft_consensus.cc:374] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:43.105132 4278 raft_consensus.cc:729] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab942440f41439e821fe3dcb8457782, State: Initialized, Role: FOLLOWER
I20260810 03:50:43.105600 4278 consensus_queue.cc:260] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab942440f41439e821fe3dcb8457782" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42319 } }
I20260810 03:50:43.105741 4278 raft_consensus.cc:388] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:50:43.105809 4278 raft_consensus.cc:482] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:50:43.105914 4278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:43.107410 4278 raft_consensus.cc:504] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab942440f41439e821fe3dcb8457782" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42319 } }
I20260810 03:50:43.107846 4278 leader_election.cc:302] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cab942440f41439e821fe3dcb8457782; no voters:
I20260810 03:50:43.108188 4278 leader_election.cc:288] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:50:43.108316 4290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:43.108556 4290 raft_consensus.cc:686] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782 [term 1 LEADER]: Becoming Leader. State: Replica: cab942440f41439e821fe3dcb8457782, State: Running, Role: LEADER
I20260810 03:50:43.108912 4290 consensus_queue.cc:237] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab942440f41439e821fe3dcb8457782" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42319 } }
I20260810 03:50:43.109521 4278 sys_catalog.cc:564] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:50:43.110707 4291 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cab942440f41439e821fe3dcb8457782" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab942440f41439e821fe3dcb8457782" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42319 } } }
I20260810 03:50:43.110854 4291 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:43.110687 4292 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cab942440f41439e821fe3dcb8457782. Latest consensus state: current_term: 1 leader_uuid: "cab942440f41439e821fe3dcb8457782" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab942440f41439e821fe3dcb8457782" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42319 } } }
I20260810 03:50:43.111097 4292 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:43.111891 4302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:50:43.115255 4302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:50:43.122766 4302 catalog_manager.cc:1347] Generated new cluster ID: 325272ac063d41f8abed8f52ea84c5ca
I20260810 03:50:43.122848 4302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:50:43.152818 4302 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:50:43.153204 4302 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:50:43.160815 4302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782: Generated new TSK 0
I20260810 03:50:43.161671 4302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:50:43.179857 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:42319
--builtin_ntp_servers=127.27.153.84:40803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 03:50:43.301905 4314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:43.302230 4314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:43.302269 4314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:43.306613 4314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:43.306701 4314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:43.306762 4314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:50:43.312177 4314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:42319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:43.313364 4314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:43.314872 4314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:50:43.325965 4314 server_base.cc:1034] running on GCE node
W20260810 03:50:43.325892 4319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:43.325882 4320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:43.326221 4322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:43.326689 4314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:43.327637 4314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:43.328851 4314 hybrid_clock.cc:648] HybridClock initialized: now 1786333843328827 us; error 37 us; skew 500 ppm
I20260810 03:50:43.330981 4314 webserver.cc:460] Webserver started at http://127.27.153.65:37581/ using document root <none> and password file <none>
I20260810 03:50:43.331703 4314 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:43.331779 4314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:43.332000 4314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:43.334463 4314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "d596bad7a66b4d0e89f448def595235b"
format_stamp: "Formatted at 2026-08-10 03:50:43 on dist-test-slave-pww0"
server_key: "85d31798b95cbf4c842d4389b87965c4"
server_key_iv: "15cd8e8d784773c5eea6b1a435fe6d12"
server_key_version: "encryptionkey@0"
I20260810 03:50:43.335114 4314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "d596bad7a66b4d0e89f448def595235b"
format_stamp: "Formatted at 2026-08-10 03:50:43 on dist-test-slave-pww0"
server_key: "85d31798b95cbf4c842d4389b87965c4"
server_key_iv: "15cd8e8d784773c5eea6b1a435fe6d12"
server_key_version: "encryptionkey@0"
I20260810 03:50:43.339425 4314 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 03:50:43.342365 4338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:43.343880 4314 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:43.344025 4314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "d596bad7a66b4d0e89f448def595235b"
format_stamp: "Formatted at 2026-08-10 03:50:43 on dist-test-slave-pww0"
server_key: "85d31798b95cbf4c842d4389b87965c4"
server_key_iv: "15cd8e8d784773c5eea6b1a435fe6d12"
server_key_version: "encryptionkey@0"
I20260810 03:50:43.344149 4314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:43.349223 4314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:43.349921 4314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:43.350104 4314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:43.350706 4314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:43.351758 4314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:43.351810 4314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:43.351866 4314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:43.351892 4314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:43.360826 4314 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:43223
I20260810 03:50:43.360857 4532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:43223 every 8 connection(s)
I20260810 03:50:43.361773 4314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:50:43.367823 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 4314
I20260810 03:50:43.367967 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:50:43.368253 28261 external_mini_cluster.cc:1428] Setting key aff93db293769566ae0769a392534fee
I20260810 03:50:43.368439 4282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:43.363660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:54749 (local address 127.27.153.126:42319)
0810 03:50:43.363786 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:43.363790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:43.364995 (+ 1205us) server_negotiation.cc:407] Connection header received
0810 03:50:43.366518 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:43.366522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:43.366576 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:43.366651 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:43.367904 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:43.367908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:43.367911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:50:43.367929 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:43.367958 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:50:43.367969 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:43.368086 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:43.368241 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 03:50:43.368293 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 03:50:43.369365 4540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:43.364083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42319 (local address 127.27.153.65:54749)
0810 03:50:43.364821 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:43.364864 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:43.366253 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:43.366850 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:43.366857 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:43.366956 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:43.367554 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:50:43.367556 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:43.367615 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:50:43.367715 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:43.368077 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:43.368087 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:43.368094 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:50:43.368285 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 03:50:43.368569 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":121,"threads_started":1}
I20260810 03:50:43.370303 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:42319
--builtin_ntp_servers=127.27.153.84:40803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 03:50:43.370661 4534 heartbeater.cc:347] Connected to a master server at 127.27.153.126:42319
I20260810 03:50:43.370934 4534 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:43.371444 4534 heartbeater.cc:511] Master 127.27.153.126:42319 requested a full tablet report, sending...
I20260810 03:50:43.373442 4208 ts_manager.cc:194] Registered new tserver with Master: d596bad7a66b4d0e89f448def595235b (127.27.153.65:43223)
W20260810 03:50:43.495831 4542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:43.496160 4542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:43.496206 4542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:43.501147 4542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:43.501240 4542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:43.501304 4542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:50:43.506594 4542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:42319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:43.507934 4542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:43.509595 4542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:43.520426 4554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:43.520426 4553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:43.520471 4542 server_base.cc:1034] running on GCE node
W20260810 03:50:43.520426 4556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:43.521205 4542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:43.521991 4542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:43.523219 4542 hybrid_clock.cc:648] HybridClock initialized: now 1786333843523199 us; error 32 us; skew 500 ppm
I20260810 03:50:43.525364 4542 webserver.cc:460] Webserver started at http://127.27.153.66:41043/ using document root <none> and password file <none>
I20260810 03:50:43.525992 4542 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:43.526064 4542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:43.526270 4542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:43.528743 4542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "2610ba1a77984ea490e44221943ae222"
format_stamp: "Formatted at 2026-08-10 03:50:43 on dist-test-slave-pww0"
server_key: "58558f86d02468fa55d9fc8e6bf30707"
server_key_iv: "bb28de1faab27f908a6791af72c5c935"
server_key_version: "encryptionkey@0"
I20260810 03:50:43.529394 4542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "2610ba1a77984ea490e44221943ae222"
format_stamp: "Formatted at 2026-08-10 03:50:43 on dist-test-slave-pww0"
server_key: "58558f86d02468fa55d9fc8e6bf30707"
server_key_iv: "bb28de1faab27f908a6791af72c5c935"
server_key_version: "encryptionkey@0"
I20260810 03:50:43.533821 4542 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:50:43.537050 4565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:43.538582 4542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:50:43.538750 4542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "2610ba1a77984ea490e44221943ae222"
format_stamp: "Formatted at 2026-08-10 03:50:43 on dist-test-slave-pww0"
server_key: "58558f86d02468fa55d9fc8e6bf30707"
server_key_iv: "bb28de1faab27f908a6791af72c5c935"
server_key_version: "encryptionkey@0"
I20260810 03:50:43.538892 4542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:43.544189 4542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:43.544978 4542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:43.545172 4542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:43.545828 4542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:43.547138 4542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:43.547201 4542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:43.547263 4542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:43.547291 4542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:43.556761 4542 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:42265
I20260810 03:50:43.556780 4762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:42265 every 8 connection(s)
I20260810 03:50:43.557808 4542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:50:43.558777 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 4542
I20260810 03:50:43.558907 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:50:43.559206 28261 external_mini_cluster.cc:1428] Setting key 727fa5acfa0e42d07ff3d6a441d92d2d
I20260810 03:50:43.561504 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:42319
--builtin_ntp_servers=127.27.153.84:40803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 03:50:43.565735 4282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:43.560089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:51587 (local address 127.27.153.126:42319)
0810 03:50:43.560254 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:43.560259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:43.561794 (+ 1535us) server_negotiation.cc:407] Connection header received
0810 03:50:43.563563 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:43.563567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:43.563623 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:43.563722 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:43.565136 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:43.565141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:43.565143 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:50:43.565166 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:43.565205 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:50:43.565217 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:43.565319 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:43.565556 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 03:50:43.565614 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:50:43.567039 4765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:43.560620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42319 (local address 127.27.153.66:51587)
0810 03:50:43.561607 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:43.561673 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:43.563219 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:43.563902 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:43.563909 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:43.564020 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:43.564748 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:50:43.564750 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:43.564797 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:50:43.564878 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:43.565347 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:43.565359 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:43.565365 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 03:50:43.565575 (+ 210us) client_negotiation.cc:240] Negotiation successful
0810 03:50:43.565968 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":156,"threads_started":1}
I20260810 03:50:43.568441 4763 heartbeater.cc:347] Connected to a master server at 127.27.153.126:42319
I20260810 03:50:43.568733 4763 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:43.569296 4763 heartbeater.cc:511] Master 127.27.153.126:42319 requested a full tablet report, sending...
I20260810 03:50:43.570605 4208 ts_manager.cc:194] Registered new tserver with Master: 2610ba1a77984ea490e44221943ae222 (127.27.153.66:42265)
W20260810 03:50:43.683211 4766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:43.683583 4766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:43.683633 4766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:43.687902 4766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:43.687981 4766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:43.688042 4766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:50:43.693166 4766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:42319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:43.694473 4766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:43.696084 4766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:43.706593 4776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:43.706586 4775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:43.706625 4783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:43.706844 4766 server_base.cc:1034] running on GCE node
I20260810 03:50:43.707501 4766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:43.708215 4766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:43.709383 4766 hybrid_clock.cc:648] HybridClock initialized: now 1786333843709357 us; error 40 us; skew 500 ppm
I20260810 03:50:43.711635 4766 webserver.cc:460] Webserver started at http://127.27.153.67:40913/ using document root <none> and password file <none>
I20260810 03:50:43.712301 4766 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:43.712383 4766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:43.712612 4766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:43.714879 4766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "ba562282d7254046b0dab3ab5f3bd9f8"
format_stamp: "Formatted at 2026-08-10 03:50:43 on dist-test-slave-pww0"
server_key: "84460401306f2c8d68591db530acc347"
server_key_iv: "4608066ebc4898d4c4acc5ddab5ba984"
server_key_version: "encryptionkey@0"
I20260810 03:50:43.715507 4766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "ba562282d7254046b0dab3ab5f3bd9f8"
format_stamp: "Formatted at 2026-08-10 03:50:43 on dist-test-slave-pww0"
server_key: "84460401306f2c8d68591db530acc347"
server_key_iv: "4608066ebc4898d4c4acc5ddab5ba984"
server_key_version: "encryptionkey@0"
I20260810 03:50:43.719870 4766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 03:50:43.722776 4791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:43.724270 4766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:43.724414 4766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "ba562282d7254046b0dab3ab5f3bd9f8"
format_stamp: "Formatted at 2026-08-10 03:50:43 on dist-test-slave-pww0"
server_key: "84460401306f2c8d68591db530acc347"
server_key_iv: "4608066ebc4898d4c4acc5ddab5ba984"
server_key_version: "encryptionkey@0"
I20260810 03:50:43.724542 4766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:43.730250 4766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:43.731038 4766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:43.731245 4766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:43.731964 4766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:43.733047 4766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:43.733099 4766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:43.733166 4766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:43.733192 4766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:43.742827 4766 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:41903
I20260810 03:50:43.742884 5012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:41903 every 8 connection(s)
I20260810 03:50:43.744019 4766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:50:43.750849 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 4766
I20260810 03:50:43.750972 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:50:43.751112 4282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:43.746158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:39879 (local address 127.27.153.126:42319)
0810 03:50:43.746318 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:43.746323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:43.747605 (+ 1282us) server_negotiation.cc:407] Connection header received
0810 03:50:43.749120 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:43.749124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:43.749193 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:43.749279 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:43.750517 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:43.750520 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:43.750522 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:50:43.750537 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:43.750583 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:50:43.750592 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:43.750697 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:43.750909 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 03:50:43.750961 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:50:43.751255 28261 external_mini_cluster.cc:1428] Setting key ae6c2e2b1a4506a74273379f1a86e96d
I20260810 03:50:43.752281 5016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:43.746635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42319 (local address 127.27.153.67:39879)
0810 03:50:43.747414 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:43.747489 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:43.748822 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:43.749464 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:43.749471 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:43.749572 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:43.750185 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:50:43.750187 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:43.750233 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:50:43.750328 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:43.750750 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:43.750760 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:43.750765 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 03:50:43.750946 (+ 181us) client_negotiation.cc:240] Negotiation successful
0810 03:50:43.751224 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":131,"threads_started":1}
I20260810 03:50:43.753484 5013 heartbeater.cc:347] Connected to a master server at 127.27.153.126:42319
I20260810 03:50:43.753758 5013 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:43.754205 5013 heartbeater.cc:511] Master 127.27.153.126:42319 requested a full tablet report, sending...
I20260810 03:50:43.755594 4208 ts_manager.cc:194] Registered new tserver with Master: ba562282d7254046b0dab3ab5f3bd9f8 (127.27.153.67:41903)
I20260810 03:50:43.766039 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:50:43.769915 4282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:43.768272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46180 (local address 127.27.153.126:42319)
0810 03:50:43.768462 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:43.768467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:43.768581 (+ 114us) server_negotiation.cc:407] Connection header received
0810 03:50:43.768727 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:43.768730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:43.768771 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:43.768834 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:43.769500 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:43.769504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:43.769507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:50:43.769519 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:43.769559 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:50:43.769567 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:43.769642 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:43.769726 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 03:50:43.769771 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 03:50:43.773571 4208 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:50:43.777195 4208 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:50:43.790202 5029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:43.787276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.27.153.65:43223)
0810 03:50:43.788046 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:43.788059 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:43.788086 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:50:43.788143 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:43.788149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:43.788381 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:43.788530 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:43.789596 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:43.789602 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:43.789616 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:50:43.789628 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:43.789676 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:50:43.789715 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:43.789834 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:43.789943 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 03:50:43.790026 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":129,"threads_started":1}
I20260810 03:50:43.790186 5028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:43.787133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:43223 (local address 127.0.0.1:58872)
0810 03:50:43.787580 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:43.787643 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:43.787774 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:43.788529 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:43.788535 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:43.788614 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:43.789355 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:50:43.789356 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:43.789377 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:50:43.789452 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:43.789797 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:43.789807 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:43.789814 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:50:43.789919 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 03:50:43.790046 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":134,"threads_started":1}
I20260810 03:50:43.791407 5030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:43.787986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:41903 (local address 127.0.0.1:39198)
0810 03:50:43.789711 (+ 1725us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:43.789726 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:43.789820 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:43.790285 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:43.790288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:43.790316 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:43.790732 (+ 416us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:50:43.790734 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:43.790748 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:50:43.790779 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:43.791108 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:43.791112 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:43.791116 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 03:50:43.791215 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 03:50:43.791258 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1664,"thread_start_us":117,"threads_started":1}
I20260810 03:50:43.792196 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:43.787152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:42265 (local address 127.0.0.1:57940)
0810 03:50:43.787727 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:43.787739 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:43.787795 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:43.789541 (+ 1746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:43.789544 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:43.789560 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:43.791533 (+ 1973us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:50:43.791535 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:43.791547 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:50:43.791576 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:43.791947 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:43.791950 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:43.791953 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 03:50:43.792039 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 03:50:43.792075 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":81,"threads_started":1}
I20260810 03:50:43.791507 5032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:43.788338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39198 (local address 127.27.153.67:41903)
0810 03:50:43.789222 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:43.789235 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:43.789923 (+ 688us) server_negotiation.cc:407] Connection header received
0810 03:50:43.789989 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:43.789994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:43.790148 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:43.790280 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:43.790940 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:43.790946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:43.790959 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:50:43.790970 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:43.791004 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:50:43.791023 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:43.791140 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:43.791244 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 03:50:43.791320 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":188,"threads_started":1}
I20260810 03:50:43.792690 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:43.788171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57940 (local address 127.27.153.66:42265)
0810 03:50:43.788505 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:43.788519 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:43.788548 (+ 29us) server_negotiation.cc:407] Connection header received
0810 03:50:43.788609 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:43.788615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:43.788791 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:43.788932 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:43.791715 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:43.791723 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:43.791736 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:50:43.791751 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:43.791801 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:50:43.791848 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:43.791967 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:43.792437 (+ 470us) server_negotiation.cc:299] Negotiation successful
0810 03:50:43.792542 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":103,"threads_started":1}
I20260810 03:50:43.793954 4422 tablet_service.cc:1467] Processing CreateTablet for tablet 9e64444f664f4aefafa9e18cafc1b4c2 (DEFAULT_TABLE table=test-table [id=25e9050df8714bbfb2466354975b5553]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:43.795208 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e64444f664f4aefafa9e18cafc1b4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:43.795233 4880 tablet_service.cc:1467] Processing CreateTablet for tablet 9e64444f664f4aefafa9e18cafc1b4c2 (DEFAULT_TABLE table=test-table [id=25e9050df8714bbfb2466354975b5553]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:43.796499 4880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e64444f664f4aefafa9e18cafc1b4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:43.797916 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 9e64444f664f4aefafa9e18cafc1b4c2 (DEFAULT_TABLE table=test-table [id=25e9050df8714bbfb2466354975b5553]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:43.799141 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e64444f664f4aefafa9e18cafc1b4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:43.801991 5034 tablet_bootstrap.cc:492] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b: Bootstrap starting.
I20260810 03:50:43.804436 5034 tablet_bootstrap.cc:654] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:43.805299 5034 log.cc:824] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:43.805349 5036 tablet_bootstrap.cc:492] T 9e64444f664f4aefafa9e18cafc1b4c2 P ba562282d7254046b0dab3ab5f3bd9f8: Bootstrap starting.
I20260810 03:50:43.807628 5034 tablet_bootstrap.cc:492] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b: No bootstrap required, opened a new log
I20260810 03:50:43.807689 5039 tablet_bootstrap.cc:492] T 9e64444f664f4aefafa9e18cafc1b4c2 P 2610ba1a77984ea490e44221943ae222: Bootstrap starting.
I20260810 03:50:43.807848 5034 ts_tablet_manager.cc:1397] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:50:43.807945 5036 tablet_bootstrap.cc:654] T 9e64444f664f4aefafa9e18cafc1b4c2 P ba562282d7254046b0dab3ab5f3bd9f8: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:43.808856 5036 log.cc:824] T 9e64444f664f4aefafa9e18cafc1b4c2 P ba562282d7254046b0dab3ab5f3bd9f8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:43.809832 5039 tablet_bootstrap.cc:654] T 9e64444f664f4aefafa9e18cafc1b4c2 P 2610ba1a77984ea490e44221943ae222: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:43.811372 5036 tablet_bootstrap.cc:492] T 9e64444f664f4aefafa9e18cafc1b4c2 P ba562282d7254046b0dab3ab5f3bd9f8: No bootstrap required, opened a new log
I20260810 03:50:43.811385 5039 log.cc:824] T 9e64444f664f4aefafa9e18cafc1b4c2 P 2610ba1a77984ea490e44221943ae222: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:43.811654 5036 ts_tablet_manager.cc:1397] T 9e64444f664f4aefafa9e18cafc1b4c2 P ba562282d7254046b0dab3ab5f3bd9f8: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260810 03:50:43.812440 5034 raft_consensus.cc:348] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d596bad7a66b4d0e89f448def595235b" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43223 } } peers { permanent_uuid: "2610ba1a77984ea490e44221943ae222" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 42265 } } peers { permanent_uuid: "ba562282d7254046b0dab3ab5f3bd9f8" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41903 } }
I20260810 03:50:43.812742 5034 raft_consensus.cc:374] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:43.812801 5034 raft_consensus.cc:729] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d596bad7a66b4d0e89f448def595235b, State: Initialized, Role: FOLLOWER
I20260810 03:50:43.813284 5034 consensus_queue.cc:260] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d596bad7a66b4d0e89f448def595235b" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43223 } } peers { permanent_uuid: "2610ba1a77984ea490e44221943ae222" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 42265 } } peers { permanent_uuid: "ba562282d7254046b0dab3ab5f3bd9f8" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41903 } }
I20260810 03:50:43.813477 5039 tablet_bootstrap.cc:492] T 9e64444f664f4aefafa9e18cafc1b4c2 P 2610ba1a77984ea490e44221943ae222: No bootstrap required, opened a new log
I20260810 03:50:43.813692 5039 ts_tablet_manager.cc:1397] T 9e64444f664f4aefafa9e18cafc1b4c2 P 2610ba1a77984ea490e44221943ae222: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:50:43.814383 5034 ts_tablet_manager.cc:1428] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 03:50:43.814823 4534 heartbeater.cc:503] Master 127.27.153.126:42319 was elected leader, sending a full tablet report...
I20260810 03:50:43.816031 5036 raft_consensus.cc:348] T 9e64444f664f4aefafa9e18cafc1b4c2 P ba562282d7254046b0dab3ab5f3bd9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d596bad7a66b4d0e89f448def595235b" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43223 } } peers { permanent_uuid: "2610ba1a77984ea490e44221943ae222" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 42265 } } peers { permanent_uuid: "ba562282d7254046b0dab3ab5f3bd9f8" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41903 } }
I20260810 03:50:43.816311 5036 raft_consensus.cc:374] T 9e64444f664f4aefafa9e18cafc1b4c2 P ba562282d7254046b0dab3ab5f3bd9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:43.816372 5036 raft_consensus.cc:729] T 9e64444f664f4aefafa9e18cafc1b4c2 P ba562282d7254046b0dab3ab5f3bd9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba562282d7254046b0dab3ab5f3bd9f8, State: Initialized, Role: FOLLOWER
I20260810 03:50:43.816838 5036 consensus_queue.cc:260] T 9e64444f664f4aefafa9e18cafc1b4c2 P ba562282d7254046b0dab3ab5f3bd9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d596bad7a66b4d0e89f448def595235b" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43223 } } peers { permanent_uuid: "2610ba1a77984ea490e44221943ae222" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 42265 } } peers { permanent_uuid: "ba562282d7254046b0dab3ab5f3bd9f8" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41903 } }
I20260810 03:50:43.817873 5013 heartbeater.cc:503] Master 127.27.153.126:42319 was elected leader, sending a full tablet report...
I20260810 03:50:43.817974 5036 ts_tablet_manager.cc:1428] T 9e64444f664f4aefafa9e18cafc1b4c2 P ba562282d7254046b0dab3ab5f3bd9f8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 03:50:43.820398 5039 raft_consensus.cc:348] T 9e64444f664f4aefafa9e18cafc1b4c2 P 2610ba1a77984ea490e44221943ae222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d596bad7a66b4d0e89f448def595235b" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43223 } } peers { permanent_uuid: "2610ba1a77984ea490e44221943ae222" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 42265 } } peers { permanent_uuid: "ba562282d7254046b0dab3ab5f3bd9f8" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41903 } }
I20260810 03:50:43.820717 5039 raft_consensus.cc:374] T 9e64444f664f4aefafa9e18cafc1b4c2 P 2610ba1a77984ea490e44221943ae222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:43.820791 5039 raft_consensus.cc:729] T 9e64444f664f4aefafa9e18cafc1b4c2 P 2610ba1a77984ea490e44221943ae222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2610ba1a77984ea490e44221943ae222, State: Initialized, Role: FOLLOWER
I20260810 03:50:43.821311 5039 consensus_queue.cc:260] T 9e64444f664f4aefafa9e18cafc1b4c2 P 2610ba1a77984ea490e44221943ae222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d596bad7a66b4d0e89f448def595235b" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43223 } } peers { permanent_uuid: "2610ba1a77984ea490e44221943ae222" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 42265 } } peers { permanent_uuid: "ba562282d7254046b0dab3ab5f3bd9f8" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41903 } }
I20260810 03:50:43.822088 4763 heartbeater.cc:503] Master 127.27.153.126:42319 was elected leader, sending a full tablet report...
I20260810 03:50:43.822469 5039 ts_tablet_manager.cc:1428] T 9e64444f664f4aefafa9e18cafc1b4c2 P 2610ba1a77984ea490e44221943ae222: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
W20260810 03:50:43.863004 4538 tablet.cc:2369] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:50:43.877876 5046 raft_consensus.cc:482] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:50:43.878142 5046 raft_consensus.cc:504] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d596bad7a66b4d0e89f448def595235b" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43223 } } peers { permanent_uuid: "2610ba1a77984ea490e44221943ae222" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 42265 } } peers { permanent_uuid: "ba562282d7254046b0dab3ab5f3bd9f8" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41903 } }
I20260810 03:50:43.879338 5046 leader_election.cc:288] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2610ba1a77984ea490e44221943ae222 (127.27.153.66:42265), ba562282d7254046b0dab3ab5f3bd9f8 (127.27.153.67:41903)
I20260810 03:50:43.881551 5032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:43.879844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:54785 (local address 127.27.153.67:41903)
0810 03:50:43.879994 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:43.879999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:43.880144 (+ 145us) server_negotiation.cc:407] Connection header received
0810 03:50:43.880272 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:43.880276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:43.880318 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:43.880399 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:43.881154 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:43.881158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:43.881160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:50:43.881174 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:43.881196 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:50:43.881207 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:43.881294 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:43.881368 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 03:50:43.881415 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 03:50:43.881573 5049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:43.879677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:41903 (local address 127.27.153.65:54785)
0810 03:50:43.880037 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:43.880054 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:43.880176 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:43.880437 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:43.880440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:43.880455 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:43.880954 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:50:43.880956 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:43.880968 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:50:43.881002 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:43.881282 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:43.881285 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:43.881288 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 03:50:43.881377 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 03:50:43.881415 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":106,"threads_started":1}
I20260810 03:50:43.881621 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:43.879926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:47319 (local address 127.27.153.66:42265)
0810 03:50:43.880096 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:43.880101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:43.880229 (+ 128us) server_negotiation.cc:407] Connection header received
0810 03:50:43.880330 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:43.880332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:43.880365 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:43.880441 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:43.881268 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:43.881272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:43.881275 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:50:43.881288 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:43.881310 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:50:43.881320 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:43.881403 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:43.881470 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 03:50:43.881505 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 03:50:43.881839 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:43.879677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:42265 (local address 127.27.153.65:47319)
0810 03:50:43.880128 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:43.880143 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:43.880245 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:43.880476 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:43.880479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:43.880498 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:43.881108 (+ 610us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:50:43.881109 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:43.881119 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:50:43.881145 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:43.881396 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:43.881398 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:43.881401 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 03:50:43.881476 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 03:50:43.881509 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":21,"thread_start_us":77,"threads_started":1}
I20260810 03:50:43.882359 4921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e64444f664f4aefafa9e18cafc1b4c2" candidate_uuid: "d596bad7a66b4d0e89f448def595235b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba562282d7254046b0dab3ab5f3bd9f8" is_pre_election: true
I20260810 03:50:43.882411 4685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e64444f664f4aefafa9e18cafc1b4c2" candidate_uuid: "d596bad7a66b4d0e89f448def595235b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2610ba1a77984ea490e44221943ae222" is_pre_election: true
I20260810 03:50:43.882670 4921 raft_consensus.cc:2454] T 9e64444f664f4aefafa9e18cafc1b4c2 P ba562282d7254046b0dab3ab5f3bd9f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d596bad7a66b4d0e89f448def595235b in term 0.
I20260810 03:50:43.882673 4685 raft_consensus.cc:2454] T 9e64444f664f4aefafa9e18cafc1b4c2 P 2610ba1a77984ea490e44221943ae222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d596bad7a66b4d0e89f448def595235b in term 0.
I20260810 03:50:43.883164 4340 leader_election.cc:302] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba562282d7254046b0dab3ab5f3bd9f8, d596bad7a66b4d0e89f448def595235b; no voters:
I20260810 03:50:43.883441 5046 raft_consensus.cc:2781] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:50:43.883565 5046 raft_consensus.cc:482] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:50:43.883616 5046 raft_consensus.cc:3037] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:43.884757 5046 raft_consensus.cc:504] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d596bad7a66b4d0e89f448def595235b" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43223 } } peers { permanent_uuid: "2610ba1a77984ea490e44221943ae222" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 42265 } } peers { permanent_uuid: "ba562282d7254046b0dab3ab5f3bd9f8" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41903 } }
I20260810 03:50:43.885100 5046 leader_election.cc:288] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b [CANDIDATE]: Term 1 election: Requested vote from peers 2610ba1a77984ea490e44221943ae222 (127.27.153.66:42265), ba562282d7254046b0dab3ab5f3bd9f8 (127.27.153.67:41903)
I20260810 03:50:43.885476 4685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e64444f664f4aefafa9e18cafc1b4c2" candidate_uuid: "d596bad7a66b4d0e89f448def595235b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2610ba1a77984ea490e44221943ae222"
I20260810 03:50:43.885504 4921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e64444f664f4aefafa9e18cafc1b4c2" candidate_uuid: "d596bad7a66b4d0e89f448def595235b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba562282d7254046b0dab3ab5f3bd9f8"
I20260810 03:50:43.885593 4685 raft_consensus.cc:3037] T 9e64444f664f4aefafa9e18cafc1b4c2 P 2610ba1a77984ea490e44221943ae222 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:43.885629 4921 raft_consensus.cc:3037] T 9e64444f664f4aefafa9e18cafc1b4c2 P ba562282d7254046b0dab3ab5f3bd9f8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:43.886699 4685 raft_consensus.cc:2454] T 9e64444f664f4aefafa9e18cafc1b4c2 P 2610ba1a77984ea490e44221943ae222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d596bad7a66b4d0e89f448def595235b in term 1.
I20260810 03:50:43.886765 4921 raft_consensus.cc:2454] T 9e64444f664f4aefafa9e18cafc1b4c2 P ba562282d7254046b0dab3ab5f3bd9f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d596bad7a66b4d0e89f448def595235b in term 1.
I20260810 03:50:43.887040 4341 leader_election.cc:302] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2610ba1a77984ea490e44221943ae222, d596bad7a66b4d0e89f448def595235b; no voters:
I20260810 03:50:43.887223 5046 raft_consensus.cc:2781] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:43.887485 5046 raft_consensus.cc:686] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b [term 1 LEADER]: Becoming Leader. State: Replica: d596bad7a66b4d0e89f448def595235b, State: Running, Role: LEADER
I20260810 03:50:43.887872 5046 consensus_queue.cc:237] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d596bad7a66b4d0e89f448def595235b" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43223 } } peers { permanent_uuid: "2610ba1a77984ea490e44221943ae222" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 42265 } } peers { permanent_uuid: "ba562282d7254046b0dab3ab5f3bd9f8" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41903 } }
I20260810 03:50:43.891348 4211 catalog_manager.cc:5495] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b reported cstate change: term changed from 0 to 1, leader changed from <none> to d596bad7a66b4d0e89f448def595235b (127.27.153.65). New cstate: current_term: 1 leader_uuid: "d596bad7a66b4d0e89f448def595235b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d596bad7a66b4d0e89f448def595235b" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2610ba1a77984ea490e44221943ae222" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 42265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba562282d7254046b0dab3ab5f3bd9f8" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41903 } health_report { overall_health: UNKNOWN } } }
I20260810 03:50:43.924316 5029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:43.922760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.27.153.65:43223)
0810 03:50:43.922906 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:43.922911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:43.922926 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:50:43.923025 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:43.923029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:43.923092 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:43.923165 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:43.923906 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:43.923910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:43.923913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:50:43.923940 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:43.923959 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:50:43.923968 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:43.924048 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:43.924129 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 03:50:43.924169 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 03:50:43.930601 4921 raft_consensus.cc:1261] T 9e64444f664f4aefafa9e18cafc1b4c2 P ba562282d7254046b0dab3ab5f3bd9f8 [term 1 FOLLOWER]: Refusing update from remote peer d596bad7a66b4d0e89f448def595235b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:43.930648 4685 raft_consensus.cc:1261] T 9e64444f664f4aefafa9e18cafc1b4c2 P 2610ba1a77984ea490e44221943ae222 [term 1 FOLLOWER]: Refusing update from remote peer d596bad7a66b4d0e89f448def595235b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:43.931416 5052 consensus_queue.cc:1035] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2610ba1a77984ea490e44221943ae222" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 42265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:43.931627 5046 consensus_queue.cc:1035] T 9e64444f664f4aefafa9e18cafc1b4c2 P d596bad7a66b4d0e89f448def595235b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba562282d7254046b0dab3ab5f3bd9f8" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:43.938902 5058 mvcc.cc:205] Tried to move back new op lower bound from 7316823424731709440 to 7316823424567398400. Current Snapshot: MvccSnapshot[applied={T|T < 7316823424731709440}]
I20260810 03:50:43.943967 5060 mvcc.cc:205] Tried to move back new op lower bound from 7316823424731709440 to 7316823424567398400. Current Snapshot: MvccSnapshot[applied={T|T < 7316823424731709440}]
I20260810 03:50:43.948516 4211 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:43.948701 4211 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:43.951169 4211 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cab942440f41439e821fe3dcb8457782: Sending DeleteTablet for 3 replicas of tablet 9e64444f664f4aefafa9e18cafc1b4c2
I20260810 03:50:43.952088 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e64444f664f4aefafa9e18cafc1b4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:43 UTC) from {username='slave'} at 127.0.0.1:57940
I20260810 03:50:43.952093 4422 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e64444f664f4aefafa9e18cafc1b4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:43 UTC) from {username='slave'} at 127.0.0.1:58872
I20260810 03:50:43.952152 4880 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e64444f664f4aefafa9e18cafc1b4c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:43 UTC) from {username='slave'} at 127.0.0.1:39198
I20260810 03:50:43.952739 5080 tablet_replica.cc:330] T 9e64444f664f4aefafa9e18cafc1b4c2 P ba562282d7254046b0dab3ab5f3bd9f8: stopping tablet replica
I20260810 03:50:43.952768 5078 tablet_replica.cc:330] T 9e64444f664f4aefafa9e18cafc1b4c2 P 2610ba1a77984ea490e44221943ae222: stopping tablet replica
I20260810 03:50:43.953114 5080 raft_consensus.cc:2229] T 9e64444f664f4aefafa9e18cafc1b4c2 P ba562282d7254046b0dab3ab5f3bd9f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:43.953114 5078 raft_consensus.cc:2229] T 9e64444f664f4aefafa9e18cafc1b4c2 P 2610ba1a77984ea490e44221943ae222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:43.953455 5080 raft_consensus.cc:2258] T 9e64444f664f4aefafa9e18cafc1b4c2 P ba562282d7254046b0dab3ab5f3bd9f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:43.953455 5078 raft_consensus.cc:2258] T 9e64444f664f4aefafa9e18cafc1b4c2 P 2610ba1a77984ea490e44221943ae222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:43.953591 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 4314
I20260810 03:50:43.973444 5080 ts_tablet_manager.cc:1905] T 9e64444f664f4aefafa9e18cafc1b4c2 P ba562282d7254046b0dab3ab5f3bd9f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:43.976405 5080 ts_tablet_manager.cc:1918] T 9e64444f664f4aefafa9e18cafc1b4c2 P ba562282d7254046b0dab3ab5f3bd9f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:50:43.976445 5078 ts_tablet_manager.cc:1905] T 9e64444f664f4aefafa9e18cafc1b4c2 P 2610ba1a77984ea490e44221943ae222: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:43.976531 5080 log.cc:1196] T 9e64444f664f4aefafa9e18cafc1b4c2 P ba562282d7254046b0dab3ab5f3bd9f8: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/9e64444f664f4aefafa9e18cafc1b4c2
I20260810 03:50:43.976830 5080 ts_tablet_manager.cc:1939] T 9e64444f664f4aefafa9e18cafc1b4c2 P ba562282d7254046b0dab3ab5f3bd9f8: Deleting consensus metadata
I20260810 03:50:43.978106 4184 catalog_manager.cc:4841] TS ba562282d7254046b0dab3ab5f3bd9f8 (127.27.153.67:41903): tablet 9e64444f664f4aefafa9e18cafc1b4c2 (table test-table [id=25e9050df8714bbfb2466354975b5553]) successfully deleted
I20260810 03:50:43.979355 5078 ts_tablet_manager.cc:1918] T 9e64444f664f4aefafa9e18cafc1b4c2 P 2610ba1a77984ea490e44221943ae222: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:50:43.979501 5078 log.cc:1196] T 9e64444f664f4aefafa9e18cafc1b4c2 P 2610ba1a77984ea490e44221943ae222: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/9e64444f664f4aefafa9e18cafc1b4c2
I20260810 03:50:43.979838 5078 ts_tablet_manager.cc:1939] T 9e64444f664f4aefafa9e18cafc1b4c2 P 2610ba1a77984ea490e44221943ae222: Deleting consensus metadata
I20260810 03:50:43.981319 4186 catalog_manager.cc:4841] TS 2610ba1a77984ea490e44221943ae222 (127.27.153.66:42265): tablet 9e64444f664f4aefafa9e18cafc1b4c2 (table test-table [id=25e9050df8714bbfb2466354975b5553]) successfully deleted
I20260810 03:50:43.981804 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 4542
W20260810 03:50:43.981822 4184 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.153.65:43223 (error 108)
I20260810 03:50:43.992576 5027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:43.992206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:43223 (local address 127.0.0.1:58890)
0810 03:50:43.992385 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:43.992481 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:43223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260810 03:50:43.992952 4184 catalog_manager.cc:4584] TS d596bad7a66b4d0e89f448def595235b (127.27.153.65:43223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9e64444f664f4aefafa9e18cafc1b4c2: Network error: Client connection negotiation failed: client connection to 127.27.153.65:43223: connect: Connection refused (error 111)
I20260810 03:50:43.997468 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 4766
I20260810 03:50:44.019666 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 4155
2026-08-10T03:50:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1159 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 03:50:44.058491 28261 mini_oidc.cc:152] Starting JWKS server
I20260810 03:50:44.061237 28261 webserver.cc:460] Webserver started at https://127.0.0.1:46087/ using document root <none> and password file <none>
I20260810 03:50:44.061470 28261 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 03:50:44.061826 28261 webserver.cc:460] Webserver started at http://127.0.0.1:38313/ using document root <none> and password file <none>
2026-08-10T03:50:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:50:44Z Disabled control of system clock
I20260810 03:50:44.080221 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:42961
--webserver_interface=127.27.153.126
--webserver_port=0
--builtin_ntp_servers=127.27.153.84:37457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:42961
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46087/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333814837068-28261-0/testchainca.cert with env {}
W20260810 03:50:44.210192 5091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:44.210533 5091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:44.210577 5091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:44.215054 5091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:50:44.215161 5091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:44.215178 5091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:44.215194 5091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:50:44.215209 5091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 03:50:44.215225 5091 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 03:50:44.215237 5091 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46087/jwks.json
I20260810 03:50:44.221235 5091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:37457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:42961
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:42961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46087/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333814837068-28261-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:44.222406 5091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:44.223436 5091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:44.233011 5099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:44.232995 5108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:44.232995 5098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:44.233594 5091 server_base.cc:1034] running on GCE node
I20260810 03:50:44.234248 5091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:44.235509 5091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:44.236725 5091 hybrid_clock.cc:648] HybridClock initialized: now 1786333844236695 us; error 49 us; skew 500 ppm
I20260810 03:50:44.239024 5091 webserver.cc:460] Webserver started at http://127.27.153.126:34313/ using document root <none> and password file <none>
I20260810 03:50:44.239873 5091 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:44.239977 5091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:44.240244 5091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:44.242612 5091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "0a10991603fd4380b9953b6726724fc3"
format_stamp: "Formatted at 2026-08-10 03:50:44 on dist-test-slave-pww0"
server_key: "27e3ced2cf107ddd83ee1db7fcc2af5e"
server_key_iv: "42a9d850954018920fd07adb35d85699"
server_key_version: "encryptionkey@0"
I20260810 03:50:44.243278 5091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "0a10991603fd4380b9953b6726724fc3"
format_stamp: "Formatted at 2026-08-10 03:50:44 on dist-test-slave-pww0"
server_key: "27e3ced2cf107ddd83ee1db7fcc2af5e"
server_key_iv: "42a9d850954018920fd07adb35d85699"
server_key_version: "encryptionkey@0"
I20260810 03:50:44.247996 5091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:50:44.251061 5116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:44.252781 5091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:44.252944 5091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "0a10991603fd4380b9953b6726724fc3"
format_stamp: "Formatted at 2026-08-10 03:50:44 on dist-test-slave-pww0"
server_key: "27e3ced2cf107ddd83ee1db7fcc2af5e"
server_key_iv: "42a9d850954018920fd07adb35d85699"
server_key_version: "encryptionkey@0"
I20260810 03:50:44.253077 5091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:44.266671 5091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:44.267546 5091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:44.267750 5091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:44.275957 5091 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:42961
I20260810 03:50:44.276021 5202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:42961 every 8 connection(s)
I20260810 03:50:44.277048 5091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:50:44.279719 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 5091
I20260810 03:50:44.279837 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:50:44.280097 28261 external_mini_cluster.cc:1428] Setting key 0dc9e4f8e53a57f7a9c4379dd6e88574
I20260810 03:50:44.280925 5206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:44.289290 5206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a10991603fd4380b9953b6726724fc3: Bootstrap starting.
I20260810 03:50:44.292371 5213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:44.281541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35236 (local address 127.27.153.126:42961)
0810 03:50:44.282325 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:44.282347 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:44.282397 (+ 50us) server_negotiation.cc:407] Connection header received
0810 03:50:44.283246 (+ 849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:44.283274 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:44.283908 (+ 634us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:44.284573 (+ 665us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:44.285108 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:44.286587 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:44.287765 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:44.288548 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:44.288732 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:44.289774 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:44.289781 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:44.289811 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:50:44.290379 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:44.290472 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:50:44.290502 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:44.290610 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:44.290815 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 03:50:44.291181 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":131,"threads_started":1}
I20260810 03:50:44.292644 5206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a10991603fd4380b9953b6726724fc3: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:44.293680 5206 log.cc:824] T 00000000000000000000000000000000 P 0a10991603fd4380b9953b6726724fc3: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:44.295902 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:50:44.296239 5206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a10991603fd4380b9953b6726724fc3: No bootstrap required, opened a new log
I20260810 03:50:44.300335 5206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a10991603fd4380b9953b6726724fc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a10991603fd4380b9953b6726724fc3" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42961 } }
I20260810 03:50:44.300621 5206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a10991603fd4380b9953b6726724fc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:44.300673 5206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a10991603fd4380b9953b6726724fc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a10991603fd4380b9953b6726724fc3, State: Initialized, Role: FOLLOWER
I20260810 03:50:44.301165 5206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a10991603fd4380b9953b6726724fc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a10991603fd4380b9953b6726724fc3" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42961 } }
I20260810 03:50:44.301311 5206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a10991603fd4380b9953b6726724fc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:50:44.301362 5206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a10991603fd4380b9953b6726724fc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:50:44.301465 5206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a10991603fd4380b9953b6726724fc3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:44.302727 5206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a10991603fd4380b9953b6726724fc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a10991603fd4380b9953b6726724fc3" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42961 } }
I20260810 03:50:44.303108 5206 leader_election.cc:302] T 00000000000000000000000000000000 P 0a10991603fd4380b9953b6726724fc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a10991603fd4380b9953b6726724fc3; no voters:
I20260810 03:50:44.303539 5206 leader_election.cc:288] T 00000000000000000000000000000000 P 0a10991603fd4380b9953b6726724fc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:50:44.303705 5216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a10991603fd4380b9953b6726724fc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:44.304014 5216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a10991603fd4380b9953b6726724fc3 [term 1 LEADER]: Becoming Leader. State: Replica: 0a10991603fd4380b9953b6726724fc3, State: Running, Role: LEADER
I20260810 03:50:44.304366 5216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a10991603fd4380b9953b6726724fc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a10991603fd4380b9953b6726724fc3" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42961 } }
I20260810 03:50:44.304901 5206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a10991603fd4380b9953b6726724fc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:50:44.306176 5217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a10991603fd4380b9953b6726724fc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a10991603fd4380b9953b6726724fc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a10991603fd4380b9953b6726724fc3" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42961 } } }
I20260810 03:50:44.306354 5217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a10991603fd4380b9953b6726724fc3 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:44.306176 5219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a10991603fd4380b9953b6726724fc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a10991603fd4380b9953b6726724fc3. Latest consensus state: current_term: 1 leader_uuid: "0a10991603fd4380b9953b6726724fc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a10991603fd4380b9953b6726724fc3" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42961 } } }
I20260810 03:50:44.306434 5219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a10991603fd4380b9953b6726724fc3 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:44.306847 5229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:50:44.310518 5229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:50:44.318141 5229 catalog_manager.cc:1347] Generated new cluster ID: baae4209b8794c528633e1bda062c5ea
I20260810 03:50:44.318239 5229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:50:44.336506 5229 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:50:44.337571 5229 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:50:44.344556 5229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a10991603fd4380b9953b6726724fc3: Generated new TSK 0
I20260810 03:50:44.345769 5229 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 03:50:44.387385 5252 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 03:50:44.387694 5252 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 03:50:44.388151 5213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:44.363287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35248 (local address 127.27.153.126:42961)
0810 03:50:44.363479 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:44.363484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:44.363657 (+ 173us) server_negotiation.cc:407] Connection header received
0810 03:50:44.363923 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:44.363926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:44.363984 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:44.364076 (+ 92us) server_negotiation.cc:226] Negotiated authn=JWT
0810 03:50:44.364371 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:44.364924 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:44.365830 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:44.366065 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:44.366168 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:44.366578 (+ 410us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 03:50:44.387620 (+ 21042us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46087/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 03:50:44.387969 (+ 349us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35248: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46087/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":70}
W20260810 03:50:44.388253 5213 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35248: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46087/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 03:50:44.389725 5251 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:50:44.363140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42961 (local address 127.0.0.1:35248)
0810 03:50:44.363564 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:44.363578 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:44.363730 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:44.364112 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:44.364116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:44.364131 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0810 03:50:44.364250 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:44.364257 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:44.365046 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:44.365050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:44.365705 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:44.365715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:44.366186 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:44.366190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:44.366293 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:44.366353 (+ 60us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 03:50:44.387990 (+ 21637us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46087/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 03:50:44.388273 (+ 283us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.153.126:42961: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46087/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":133,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.153.126:42961: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46087/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 03:50:44.390710 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 5091
2026-08-10T03:50:44Z chronyd exiting
I20260810 03:50:44.446030 28261 test_util.cc:175] -----------------------------------------------
I20260810 03:50:44.446128 28261 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786333814837068-28261-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (390 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 03:50:44.447306 28261 mini_oidc.cc:152] Starting JWKS server
I20260810 03:50:44.448256 28261 webserver.cc:460] Webserver started at https://127.0.0.1:36999/ using document root <none> and password file <none>
I20260810 03:50:44.448359 28261 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 03:50:44.448592 28261 webserver.cc:460] Webserver started at http://127.0.0.1:37703/ using document root <none> and password file <none>
2026-08-10T03:50:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:50:44Z Disabled control of system clock
I20260810 03:50:44.459566 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:37615
--webserver_interface=127.27.153.126
--webserver_port=0
--builtin_ntp_servers=127.27.153.84:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:37615
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36999/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333814837068-28261-0/testchainca.cert with env {}
W20260810 03:50:44.581368 5261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:44.581691 5261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:44.581733 5261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:44.585963 5261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:50:44.586040 5261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:44.586053 5261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:44.586071 5261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:50:44.586084 5261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 03:50:44.586100 5261 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 03:50:44.586112 5261 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36999/jwks.json
I20260810 03:50:44.592495 5261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:37615
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:37615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36999/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333814837068-28261-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:44.593736 5261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:44.594928 5261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:44.604612 5268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:44.604650 5271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:44.604683 5269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:44.604879 5261 server_base.cc:1034] running on GCE node
I20260810 03:50:44.605571 5261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:44.606801 5261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:44.607975 5261 hybrid_clock.cc:648] HybridClock initialized: now 1786333844607946 us; error 41 us; skew 500 ppm
I20260810 03:50:44.610061 5261 webserver.cc:460] Webserver started at http://127.27.153.126:34025/ using document root <none> and password file <none>
I20260810 03:50:44.610687 5261 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:44.610759 5261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:44.610967 5261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:44.613224 5261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "b1d23c54544a42bf868f14de6ac08909"
format_stamp: "Formatted at 2026-08-10 03:50:44 on dist-test-slave-pww0"
server_key: "257e007bc5b996d8d0bae64645b8b8ac"
server_key_iv: "869bfb1558902f4e2840263e27a209a4"
server_key_version: "encryptionkey@0"
I20260810 03:50:44.613847 5261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "b1d23c54544a42bf868f14de6ac08909"
format_stamp: "Formatted at 2026-08-10 03:50:44 on dist-test-slave-pww0"
server_key: "257e007bc5b996d8d0bae64645b8b8ac"
server_key_iv: "869bfb1558902f4e2840263e27a209a4"
server_key_version: "encryptionkey@0"
I20260810 03:50:44.618032 5261 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:50:44.620899 5285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:44.622418 5261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:50:44.622578 5261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "b1d23c54544a42bf868f14de6ac08909"
format_stamp: "Formatted at 2026-08-10 03:50:44 on dist-test-slave-pww0"
server_key: "257e007bc5b996d8d0bae64645b8b8ac"
server_key_iv: "869bfb1558902f4e2840263e27a209a4"
server_key_version: "encryptionkey@0"
I20260810 03:50:44.622735 5261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:44.645112 5261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:44.645938 5261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:44.646144 5261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:44.653995 5261 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:37615
I20260810 03:50:44.654047 5373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:37615 every 8 connection(s)
I20260810 03:50:44.655076 5261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:50:44.657097 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 5261
I20260810 03:50:44.657233 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:50:44.657546 28261 external_mini_cluster.cc:1428] Setting key 0f542a51ef93bcf2fa90cc6c6f929286
I20260810 03:50:44.659219 5374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:44.665108 5374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1d23c54544a42bf868f14de6ac08909: Bootstrap starting.
I20260810 03:50:44.667038 5377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:44.659055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43452 (local address 127.27.153.126:37615)
0810 03:50:44.659824 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:44.659843 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:44.659881 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:50:44.660566 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:44.660584 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:44.661006 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:44.661339 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:44.661803 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:44.662795 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:44.663748 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:44.664245 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:44.664403 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:44.665274 (+ 871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:44.665280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:44.665294 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:50:44.665698 (+ 404us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:44.665757 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:50:44.665779 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:44.665866 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:44.666006 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 03:50:44.666288 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":165,"threads_started":1}
I20260810 03:50:44.667591 5374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1d23c54544a42bf868f14de6ac08909: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:44.668350 5374 log.cc:824] T 00000000000000000000000000000000 P b1d23c54544a42bf868f14de6ac08909: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:44.670188 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:50:44.670694 5374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1d23c54544a42bf868f14de6ac08909: No bootstrap required, opened a new log
I20260810 03:50:44.674786 5374 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1d23c54544a42bf868f14de6ac08909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1d23c54544a42bf868f14de6ac08909" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37615 } }
I20260810 03:50:44.675108 5374 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1d23c54544a42bf868f14de6ac08909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:44.675180 5374 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1d23c54544a42bf868f14de6ac08909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1d23c54544a42bf868f14de6ac08909, State: Initialized, Role: FOLLOWER
I20260810 03:50:44.675786 5374 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1d23c54544a42bf868f14de6ac08909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1d23c54544a42bf868f14de6ac08909" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37615 } }
I20260810 03:50:44.675958 5374 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1d23c54544a42bf868f14de6ac08909 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:50:44.676034 5374 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1d23c54544a42bf868f14de6ac08909 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:50:44.676149 5374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1d23c54544a42bf868f14de6ac08909 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:44.677320 5374 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1d23c54544a42bf868f14de6ac08909 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1d23c54544a42bf868f14de6ac08909" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37615 } }
I20260810 03:50:44.677695 5374 leader_election.cc:302] T 00000000000000000000000000000000 P b1d23c54544a42bf868f14de6ac08909 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1d23c54544a42bf868f14de6ac08909; no voters:
I20260810 03:50:44.678053 5374 leader_election.cc:288] T 00000000000000000000000000000000 P b1d23c54544a42bf868f14de6ac08909 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:50:44.678222 5382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1d23c54544a42bf868f14de6ac08909 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:44.678440 5382 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1d23c54544a42bf868f14de6ac08909 [term 1 LEADER]: Becoming Leader. State: Replica: b1d23c54544a42bf868f14de6ac08909, State: Running, Role: LEADER
I20260810 03:50:44.678789 5382 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1d23c54544a42bf868f14de6ac08909 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1d23c54544a42bf868f14de6ac08909" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37615 } }
I20260810 03:50:44.679395 5374 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1d23c54544a42bf868f14de6ac08909 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:50:44.680680 5383 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1d23c54544a42bf868f14de6ac08909 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1d23c54544a42bf868f14de6ac08909" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1d23c54544a42bf868f14de6ac08909" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37615 } } }
I20260810 03:50:44.680708 5384 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1d23c54544a42bf868f14de6ac08909 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1d23c54544a42bf868f14de6ac08909. Latest consensus state: current_term: 1 leader_uuid: "b1d23c54544a42bf868f14de6ac08909" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1d23c54544a42bf868f14de6ac08909" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37615 } } }
I20260810 03:50:44.680833 5383 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1d23c54544a42bf868f14de6ac08909 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:44.680833 5384 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1d23c54544a42bf868f14de6ac08909 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:44.681275 5395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:50:44.685146 5395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:50:44.691571 5395 catalog_manager.cc:1347] Generated new cluster ID: a135fd588bed4340987ff2370737ef3e
I20260810 03:50:44.691673 5395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:50:44.707329 5395 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:50:44.708721 5395 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:50:44.720783 5395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1d23c54544a42bf868f14de6ac08909: Generated new TSK 0
I20260810 03:50:44.721774 5395 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 03:50:44.751178 5423 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 03:50:44.751272 5423 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 03:50:44.752174 5377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:44.735187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43466 (local address 127.27.153.126:37615)
0810 03:50:44.735335 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:44.735340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:44.735565 (+ 225us) server_negotiation.cc:407] Connection header received
0810 03:50:44.735744 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:44.735748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:44.735818 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:44.735943 (+ 125us) server_negotiation.cc:226] Negotiated authn=JWT
0810 03:50:44.736169 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:44.737026 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:44.737969 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:44.738362 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:44.738477 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:44.738778 (+ 301us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 03:50:44.751694 (+ 12916us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36999/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 03:50:44.752007 (+ 313us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43466: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36999/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 03:50:44.752254 5377 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43466: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36999/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 03:50:44.752331 5422 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:50:44.735073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:37615 (local address 127.0.0.1:43466)
0810 03:50:44.735434 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:44.735448 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:44.735609 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:44.735943 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:44.735946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:44.735951 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0810 03:50:44.736066 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:44.736071 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:44.737177 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:44.737179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:44.737848 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:44.737858 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:44.738492 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:44.738495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:44.738563 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:44.738592 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 03:50:44.751954 (+ 13362us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36999/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 03:50:44.752135 (+ 181us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.153.126:37615: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36999/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":125,"threads_started":1}
I20260810 03:50:44.753281 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 5261
2026-08-10T03:50:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (371 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 03:50:44.819284 28261 mini_oidc.cc:152] Starting JWKS server
I20260810 03:50:44.820245 28261 webserver.cc:460] Webserver started at https://127.0.0.1:37229/ using document root <none> and password file <none>
I20260810 03:50:44.820335 28261 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 03:50:44.820550 28261 webserver.cc:460] Webserver started at http://127.0.0.1:41301/ using document root <none> and password file <none>
2026-08-10T03:50:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:50:44Z Disabled control of system clock
I20260810 03:50:44.831926 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:37759
--webserver_interface=127.27.153.126
--webserver_port=0
--builtin_ntp_servers=127.27.153.84:36007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:37759
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37229/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 03:50:44.951530 5433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:44.951859 5433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:44.951926 5433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:44.956270 5433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:50:44.956368 5433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:44.956386 5433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:44.956405 5433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:50:44.956425 5433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 03:50:44.956451 5433 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 03:50:44.956486 5433 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37229/jwks.json
I20260810 03:50:44.961880 5433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:36007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:37759
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:37759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37229/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:44.963140 5433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:44.964730 5433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:44.974259 5449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:44.974406 5433 server_base.cc:1034] running on GCE node
W20260810 03:50:44.974337 5448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:44.974340 5451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:44.975212 5433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:44.976517 5433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:44.977741 5433 hybrid_clock.cc:648] HybridClock initialized: now 1786333844977720 us; error 36 us; skew 500 ppm
I20260810 03:50:44.979957 5433 webserver.cc:460] Webserver started at http://127.27.153.126:38885/ using document root <none> and password file <none>
I20260810 03:50:44.980587 5433 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:44.980674 5433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:44.980904 5433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:44.983161 5433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "e90b7248a4fa4cdf800a57ae71e4beff"
format_stamp: "Formatted at 2026-08-10 03:50:44 on dist-test-slave-pww0"
server_key: "bd0d16d0128c480ff25f85180b999ffc"
server_key_iv: "8780daccbb05cfea1b2f106324063dec"
server_key_version: "encryptionkey@0"
I20260810 03:50:44.983827 5433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "e90b7248a4fa4cdf800a57ae71e4beff"
format_stamp: "Formatted at 2026-08-10 03:50:44 on dist-test-slave-pww0"
server_key: "bd0d16d0128c480ff25f85180b999ffc"
server_key_iv: "8780daccbb05cfea1b2f106324063dec"
server_key_version: "encryptionkey@0"
I20260810 03:50:44.988130 5433 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:50:44.990983 5459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:44.992486 5433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:44.992635 5433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "e90b7248a4fa4cdf800a57ae71e4beff"
format_stamp: "Formatted at 2026-08-10 03:50:44 on dist-test-slave-pww0"
server_key: "bd0d16d0128c480ff25f85180b999ffc"
server_key_iv: "8780daccbb05cfea1b2f106324063dec"
server_key_version: "encryptionkey@0"
I20260810 03:50:44.992782 5433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:45.022970 5433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:45.023883 5433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:45.024109 5433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:45.031754 5433 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:37759
I20260810 03:50:45.031790 5540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:37759 every 8 connection(s)
I20260810 03:50:45.032821 5433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:50:45.036345 5542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:45.040499 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 5433
I20260810 03:50:45.040639 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:50:45.040951 28261 external_mini_cluster.cc:1428] Setting key 97273cfa38a66225d875af3221b3b5d6
I20260810 03:50:45.043309 5542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90b7248a4fa4cdf800a57ae71e4beff: Bootstrap starting.
I20260810 03:50:45.045769 5542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e90b7248a4fa4cdf800a57ae71e4beff: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:45.046599 5542 log.cc:824] T 00000000000000000000000000000000 P e90b7248a4fa4cdf800a57ae71e4beff: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:45.048810 5542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90b7248a4fa4cdf800a57ae71e4beff: No bootstrap required, opened a new log
I20260810 03:50:45.050904 5550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:45.042497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52060 (local address 127.27.153.126:37759)
0810 03:50:45.043265 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:45.043282 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:45.043322 (+ 40us) server_negotiation.cc:407] Connection header received
0810 03:50:45.044078 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:45.044100 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:45.044544 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:45.044922 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:45.045412 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:45.046510 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:45.047521 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:45.048039 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:45.048198 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:45.049074 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:45.049080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:45.049094 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:50:45.049506 (+ 412us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:45.049568 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:50:45.049589 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:45.049686 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:45.049823 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 03:50:45.050104 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":172,"threads_started":1}
I20260810 03:50:45.052862 5542 raft_consensus.cc:348] T 00000000000000000000000000000000 P e90b7248a4fa4cdf800a57ae71e4beff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90b7248a4fa4cdf800a57ae71e4beff" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37759 } }
I20260810 03:50:45.053164 5542 raft_consensus.cc:374] T 00000000000000000000000000000000 P e90b7248a4fa4cdf800a57ae71e4beff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:45.053227 5542 raft_consensus.cc:729] T 00000000000000000000000000000000 P e90b7248a4fa4cdf800a57ae71e4beff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90b7248a4fa4cdf800a57ae71e4beff, State: Initialized, Role: FOLLOWER
I20260810 03:50:45.053750 5542 consensus_queue.cc:260] T 00000000000000000000000000000000 P e90b7248a4fa4cdf800a57ae71e4beff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90b7248a4fa4cdf800a57ae71e4beff" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37759 } }
I20260810 03:50:45.053925 5542 raft_consensus.cc:388] T 00000000000000000000000000000000 P e90b7248a4fa4cdf800a57ae71e4beff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:50:45.053992 5542 raft_consensus.cc:482] T 00000000000000000000000000000000 P e90b7248a4fa4cdf800a57ae71e4beff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:50:45.054111 5542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e90b7248a4fa4cdf800a57ae71e4beff [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:45.054244 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:50:45.055207 5542 raft_consensus.cc:504] T 00000000000000000000000000000000 P e90b7248a4fa4cdf800a57ae71e4beff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90b7248a4fa4cdf800a57ae71e4beff" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37759 } }
I20260810 03:50:45.055620 5542 leader_election.cc:302] T 00000000000000000000000000000000 P e90b7248a4fa4cdf800a57ae71e4beff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e90b7248a4fa4cdf800a57ae71e4beff; no voters:
I20260810 03:50:45.056039 5542 leader_election.cc:288] T 00000000000000000000000000000000 P e90b7248a4fa4cdf800a57ae71e4beff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:50:45.056180 5554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e90b7248a4fa4cdf800a57ae71e4beff [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:45.056417 5554 raft_consensus.cc:686] T 00000000000000000000000000000000 P e90b7248a4fa4cdf800a57ae71e4beff [term 1 LEADER]: Becoming Leader. State: Replica: e90b7248a4fa4cdf800a57ae71e4beff, State: Running, Role: LEADER
I20260810 03:50:45.056774 5554 consensus_queue.cc:237] T 00000000000000000000000000000000 P e90b7248a4fa4cdf800a57ae71e4beff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90b7248a4fa4cdf800a57ae71e4beff" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37759 } }
I20260810 03:50:45.057375 5542 sys_catalog.cc:564] T 00000000000000000000000000000000 P e90b7248a4fa4cdf800a57ae71e4beff [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:50:45.058465 5559 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90b7248a4fa4cdf800a57ae71e4beff [sys.catalog]: SysCatalogTable state changed. Reason: New leader e90b7248a4fa4cdf800a57ae71e4beff. Latest consensus state: current_term: 1 leader_uuid: "e90b7248a4fa4cdf800a57ae71e4beff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90b7248a4fa4cdf800a57ae71e4beff" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37759 } } }
I20260810 03:50:45.058609 5559 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90b7248a4fa4cdf800a57ae71e4beff [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:45.059026 5570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:50:45.059988 5555 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90b7248a4fa4cdf800a57ae71e4beff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e90b7248a4fa4cdf800a57ae71e4beff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90b7248a4fa4cdf800a57ae71e4beff" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37759 } } }
I20260810 03:50:45.060087 5555 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90b7248a4fa4cdf800a57ae71e4beff [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:45.062618 5570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:50:45.069090 5570 catalog_manager.cc:1347] Generated new cluster ID: dbcbc1f743324969aa71be42947a857e
I20260810 03:50:45.069190 5570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:50:45.075402 5570 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:50:45.076367 5570 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:50:45.087136 5570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e90b7248a4fa4cdf800a57ae71e4beff: Generated new TSK 0
I20260810 03:50:45.087971 5570 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:50:45.104493 5550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:45.098822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52066 (local address 127.27.153.126:37759)
0810 03:50:45.098966 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:45.098972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:45.099131 (+ 159us) server_negotiation.cc:407] Connection header received
0810 03:50:45.099303 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:45.099306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:45.099375 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:45.099543 (+ 168us) server_negotiation.cc:226] Negotiated authn=JWT
0810 03:50:45.099780 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:45.100636 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:45.101542 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:45.101911 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:45.102018 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:45.102400 (+ 382us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 03:50:45.104080 (+ 1680us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37229/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 03:50:45.104340 (+ 260us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52066: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37229/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 03:50:45.104579 5550 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52066: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37229/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 03:50:45.106074 5591 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:50:45.098715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:37759 (local address 127.0.0.1:52066)
0810 03:50:45.099042 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:45.099056 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:45.099174 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:45.099532 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:45.099534 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:45.099541 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0810 03:50:45.099669 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:45.099675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:45.100777 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:45.100780 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:45.101430 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:45.101439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:45.102070 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:45.102074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:45.102165 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:45.102194 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 03:50:45.104263 (+ 2069us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37229/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 03:50:45.105891 (+ 1628us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.153.126:37759: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37229/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":1402,"thread_start_us":103,"threads_started":1}
I20260810 03:50:45.106707 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 5433
E20260810 03:50:45.116479 5592 webserver.cc:533] Webserver: sslize failed:
E20260810 03:50:45.116557 5592 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T03:50:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (349 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T03:50:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:50:45Z Disabled control of system clock
I20260810 03:50:45.178323 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:33199
--webserver_interface=127.27.153.126
--webserver_port=0
--builtin_ntp_servers=127.27.153.84:45069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:33199
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333814837068-28261-0/testchainca.cert with env {}
W20260810 03:50:45.297040 5597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:45.297344 5597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:45.297386 5597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:45.301649 5597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:50:45.301739 5597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:45.301755 5597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:45.301772 5597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:50:45.301786 5597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 03:50:45.301801 5597 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 03:50:45.301820 5597 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 03:50:45.307605 5597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:45069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:33199
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:33199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333814837068-28261-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:45.308724 5597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:45.309722 5597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:45.318681 5605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:45.318742 5597 server_base.cc:1034] running on GCE node
W20260810 03:50:45.318647 5606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:45.318681 5608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:45.319526 5597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:45.320675 5597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:45.321825 5597 hybrid_clock.cc:648] HybridClock initialized: now 1786333845321802 us; error 39 us; skew 500 ppm
I20260810 03:50:45.323875 5597 webserver.cc:460] Webserver started at http://127.27.153.126:42127/ using document root <none> and password file <none>
I20260810 03:50:45.324493 5597 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:45.324558 5597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:45.324776 5597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:45.326908 5597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "8207555ab78041239a149e26c5b9d642"
format_stamp: "Formatted at 2026-08-10 03:50:45 on dist-test-slave-pww0"
server_key: "0e99646c8d43c8b17177fd0f31716ef6"
server_key_iv: "52952922bff05d6450cf156de2dc64af"
server_key_version: "encryptionkey@0"
I20260810 03:50:45.327538 5597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "8207555ab78041239a149e26c5b9d642"
format_stamp: "Formatted at 2026-08-10 03:50:45 on dist-test-slave-pww0"
server_key: "0e99646c8d43c8b17177fd0f31716ef6"
server_key_iv: "52952922bff05d6450cf156de2dc64af"
server_key_version: "encryptionkey@0"
I20260810 03:50:45.331578 5597 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:50:45.334189 5619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:45.335515 5597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:50:45.335651 5597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "8207555ab78041239a149e26c5b9d642"
format_stamp: "Formatted at 2026-08-10 03:50:45 on dist-test-slave-pww0"
server_key: "0e99646c8d43c8b17177fd0f31716ef6"
server_key_iv: "52952922bff05d6450cf156de2dc64af"
server_key_version: "encryptionkey@0"
I20260810 03:50:45.335765 5597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:45.346383 5597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:45.347191 5597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:45.347386 5597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:45.354728 5597 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:33199
I20260810 03:50:45.354777 5705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:33199 every 8 connection(s)
I20260810 03:50:45.355782 5597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:50:45.358702 5706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:45.364665 5706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8207555ab78041239a149e26c5b9d642: Bootstrap starting.
I20260810 03:50:45.365101 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 5597
I20260810 03:50:45.365237 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:50:45.365484 28261 external_mini_cluster.cc:1428] Setting key 24b34e46a769e29b5b5dd7251b5b44dc
I20260810 03:50:45.367403 5706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8207555ab78041239a149e26c5b9d642: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:45.368207 5706 log.cc:824] T 00000000000000000000000000000000 P 8207555ab78041239a149e26c5b9d642: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:45.370348 5706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8207555ab78041239a149e26c5b9d642: No bootstrap required, opened a new log
I20260810 03:50:45.374306 5706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8207555ab78041239a149e26c5b9d642 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8207555ab78041239a149e26c5b9d642" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33199 } }
I20260810 03:50:45.374608 5706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8207555ab78041239a149e26c5b9d642 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:45.374676 5706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8207555ab78041239a149e26c5b9d642 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8207555ab78041239a149e26c5b9d642, State: Initialized, Role: FOLLOWER
I20260810 03:50:45.375141 5706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8207555ab78041239a149e26c5b9d642 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8207555ab78041239a149e26c5b9d642" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33199 } }
I20260810 03:50:45.375269 5706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8207555ab78041239a149e26c5b9d642 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:50:45.375332 5706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8207555ab78041239a149e26c5b9d642 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:50:45.375378 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:45.366971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37080 (local address 127.27.153.126:33199)
0810 03:50:45.367704 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:45.367719 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:45.367757 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:50:45.368465 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:45.368487 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:45.368864 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:45.369198 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:45.369664 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:45.370723 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:45.371708 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:45.372204 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:45.372357 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:45.373317 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:45.373322 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:45.373340 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:50:45.373874 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:45.373955 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:50:45.373981 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:45.374076 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:45.374205 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 03:50:45.374492 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":133,"threads_started":1}
I20260810 03:50:45.375430 5706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8207555ab78041239a149e26c5b9d642 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:45.376422 5706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8207555ab78041239a149e26c5b9d642 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8207555ab78041239a149e26c5b9d642" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33199 } }
I20260810 03:50:45.376812 5706 leader_election.cc:302] T 00000000000000000000000000000000 P 8207555ab78041239a149e26c5b9d642 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8207555ab78041239a149e26c5b9d642; no voters:
I20260810 03:50:45.377174 5706 leader_election.cc:288] T 00000000000000000000000000000000 P 8207555ab78041239a149e26c5b9d642 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:50:45.377297 5713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8207555ab78041239a149e26c5b9d642 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:45.377552 5713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8207555ab78041239a149e26c5b9d642 [term 1 LEADER]: Becoming Leader. State: Replica: 8207555ab78041239a149e26c5b9d642, State: Running, Role: LEADER
I20260810 03:50:45.377869 5713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8207555ab78041239a149e26c5b9d642 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8207555ab78041239a149e26c5b9d642" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33199 } }
I20260810 03:50:45.378487 5706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8207555ab78041239a149e26c5b9d642 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:50:45.378777 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:50:45.379673 5714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8207555ab78041239a149e26c5b9d642 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8207555ab78041239a149e26c5b9d642" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8207555ab78041239a149e26c5b9d642" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33199 } } }
I20260810 03:50:45.379689 5715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8207555ab78041239a149e26c5b9d642 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8207555ab78041239a149e26c5b9d642. Latest consensus state: current_term: 1 leader_uuid: "8207555ab78041239a149e26c5b9d642" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8207555ab78041239a149e26c5b9d642" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33199 } } }
I20260810 03:50:45.379828 5714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8207555ab78041239a149e26c5b9d642 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:45.379828 5715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8207555ab78041239a149e26c5b9d642 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:45.380832 5725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:50:45.383729 5725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:50:45.389564 5725 catalog_manager.cc:1347] Generated new cluster ID: 20359922a13245e487aca1f25b639045
I20260810 03:50:45.389654 5725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:50:45.395725 5725 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:50:45.396680 5725 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:50:45.401232 5725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8207555ab78041239a149e26c5b9d642: Generated new TSK 0
I20260810 03:50:45.401962 5725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:50:45.891366 5710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:45.416220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37088 (local address 127.27.153.126:33199)
0810 03:50:45.416364 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:45.416368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:45.416592 (+ 224us) server_negotiation.cc:407] Connection header received
0810 03:50:45.416779 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:45.416782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:45.416835 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:45.416933 (+ 98us) server_negotiation.cc:226] Negotiated authn=JWT
0810 03:50:45.417183 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:45.417746 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:45.418641 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:45.418885 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:45.418980 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:45.419316 (+ 336us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 03:50:45.890947 (+471631us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 03:50:45.891231 (+ 284us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37088: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 03:50:45.891559 5710 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37088: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 03:50:45.891739 5752 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:50:45.416098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:33199 (local address 127.0.0.1:37088)
0810 03:50:45.416498 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:45.416510 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:45.416631 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:45.416943 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:45.416946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:45.416954 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0810 03:50:45.417066 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:45.417072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:45.417857 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:45.417860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:45.418514 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:45.418523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:45.419011 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:45.419015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:45.419100 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:45.419125 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 03:50:45.891208 (+472083us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 03:50:45.891519 (+ 311us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.153.126:33199: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":87,"threads_started":1}
I20260810 03:50:45.892423 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 5597
2026-08-10T03:50:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (755 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (135 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (133 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:50:46 dist-test-slave-pww0 krb5kdc[5781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:50:46 dist-test-slave-pww0 krb5kdc[5781](info): set up 2 sockets
Aug 10 03:50:46 dist-test-slave-pww0 krb5kdc[5781](info): commencing operation
krb5kdc: starting...
W20260810 03:50:47.384555 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.191s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:50:47 dist-test-slave-pww0 krb5kdc[5781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:50:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:50:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:50:47.529551 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:34961
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:34961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:47.652456 5809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:47.652839 5809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:47.652907 5809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:47.657948 5809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:50:47.658054 5809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:47.658089 5809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:47.658111 5809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:50:47.658133 5809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:50:47.664785 5809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:34961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:34961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:47.666160 5809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:47.667383 5809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:47.679018 5818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:47.679018 5817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:47.679018 5821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:47.679592 5809 server_base.cc:1034] running on GCE node
I20260810 03:50:47.680750 5809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:47.682118 5809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:47.683363 5809 hybrid_clock.cc:648] HybridClock initialized: now 1786333847683328 us; error 53 us; skew 500 ppm
Aug 10 03:50:47 dist-test-slave-pww0 krb5kdc[5781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333847, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:47.687254 5809 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:50:47.688709 5809 webserver.cc:460] Webserver started at http://127.27.153.126:32987/ using document root <none> and password file <none>
I20260810 03:50:47.689342 5809 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:47.689421 5809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:47.689646 5809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:47.692237 5809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "096229e1a3ec4dc58deb1444eeadb0a8"
format_stamp: "Formatted at 2026-08-10 03:50:47 on dist-test-slave-pww0"
server_key: "0ae731ad7dbc5a75907623bef5ec38a6"
server_key_iv: "0d676ad86729f840ed7d7749750de123"
server_key_version: "encryptionkey@0"
I20260810 03:50:47.692955 5809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "096229e1a3ec4dc58deb1444eeadb0a8"
format_stamp: "Formatted at 2026-08-10 03:50:47 on dist-test-slave-pww0"
server_key: "0ae731ad7dbc5a75907623bef5ec38a6"
server_key_iv: "0d676ad86729f840ed7d7749750de123"
server_key_version: "encryptionkey@0"
I20260810 03:50:47.697461 5809 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:50:47.700385 5835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:47.702004 5809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:50:47.702160 5809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "096229e1a3ec4dc58deb1444eeadb0a8"
format_stamp: "Formatted at 2026-08-10 03:50:47 on dist-test-slave-pww0"
server_key: "0ae731ad7dbc5a75907623bef5ec38a6"
server_key_iv: "0d676ad86729f840ed7d7749750de123"
server_key_version: "encryptionkey@0"
I20260810 03:50:47.702288 5809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:47.724711 5809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:47.728365 5809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:47.728577 5809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:47.736300 5809 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:34961
I20260810 03:50:47.736322 5930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:34961 every 8 connection(s)
I20260810 03:50:47.737406 5809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:50:47.738628 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 5809
I20260810 03:50:47.738756 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:50:47.739068 28261 external_mini_cluster.cc:1428] Setting key 20cd1b875796705fba5c0994dfc6128c
I20260810 03:50:47.741256 5932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:47.747898 5932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8: Bootstrap starting.
Aug 10 03:50:47 dist-test-slave-pww0 krb5kdc[5781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:47.750859 5932 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:47.751715 5932 log.cc:824] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:47.754810 5932 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8: No bootstrap required, opened a new log
I20260810 03:50:47.755812 5937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:47.740767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57916 (local address 127.27.153.126:34961)
0810 03:50:47.741304 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:47.741324 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:47.741377 (+ 53us) server_negotiation.cc:407] Connection header received
0810 03:50:47.742080 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:47.742099 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:47.742482 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:47.742839 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:47.743777 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:47.744852 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:47.745763 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:47.746327 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:47.746489 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:47.749545 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:47.749550 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:47.749567 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:47.749606 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:47.751428 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:47.752352 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:47.752358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:47.752363 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:47.752505 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:47.753065 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:47.753069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:47.753070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:47.753759 (+ 689us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:47.753996 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:47.754334 (+ 338us) server_negotiation.cc:299] Negotiation successful
0810 03:50:47.754700 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":354,"thread_start_us":168,"threads_started":1}
I20260810 03:50:47.759724 5932 raft_consensus.cc:348] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096229e1a3ec4dc58deb1444eeadb0a8" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 34961 } }
I20260810 03:50:47.760046 5932 raft_consensus.cc:374] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:47.760119 5932 raft_consensus.cc:729] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 096229e1a3ec4dc58deb1444eeadb0a8, State: Initialized, Role: FOLLOWER
I20260810 03:50:47.760627 5932 consensus_queue.cc:260] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096229e1a3ec4dc58deb1444eeadb0a8" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 34961 } }
I20260810 03:50:47.760797 5932 raft_consensus.cc:388] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:50:47.760860 5932 raft_consensus.cc:482] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:50:47.760965 5932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:47.762034 5932 raft_consensus.cc:504] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096229e1a3ec4dc58deb1444eeadb0a8" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 34961 } }
I20260810 03:50:47.762413 5932 leader_election.cc:302] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 096229e1a3ec4dc58deb1444eeadb0a8; no voters:
I20260810 03:50:47.762775 5932 leader_election.cc:288] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:50:47.762962 5941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:47.763228 5941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8 [term 1 LEADER]: Becoming Leader. State: Replica: 096229e1a3ec4dc58deb1444eeadb0a8, State: Running, Role: LEADER
I20260810 03:50:47.763656 5941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096229e1a3ec4dc58deb1444eeadb0a8" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 34961 } }
I20260810 03:50:47.764307 5932 sys_catalog.cc:564] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:50:47.766908 5943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "096229e1a3ec4dc58deb1444eeadb0a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096229e1a3ec4dc58deb1444eeadb0a8" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 34961 } } }
I20260810 03:50:47.766952 5945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 096229e1a3ec4dc58deb1444eeadb0a8. Latest consensus state: current_term: 1 leader_uuid: "096229e1a3ec4dc58deb1444eeadb0a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "096229e1a3ec4dc58deb1444eeadb0a8" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 34961 } } }
I20260810 03:50:47.767081 5945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:47.767079 5943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8 [sys.catalog]: This master's current role is: LEADER
W20260810 03:50:47.770035 5964 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:50:47.770130 5964 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:50:47.770216 5965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:50:47.771064 5965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:50:47.777752 5965 catalog_manager.cc:1347] Generated new cluster ID: 8cb456f910274cab9839d03279ea62e8
I20260810 03:50:47.777860 5965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:50:47.800685 5965 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:50:47.801797 5965 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:50:47.809502 5965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 096229e1a3ec4dc58deb1444eeadb0a8: Generated new TSK 0
I20260810 03:50:47.810416 5965 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.153.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
I20260810 03:50:47.855743 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:34961
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:47.977396 5975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:47.977762 5975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:47.977813 5975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:47.982520 5975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:47.982637 5975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:47.982736 5975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:50:47.989195 5975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:34961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:47.990506 5975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:47.991724 5975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:48.003916 5982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:48.003924 5987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:48.003916 5983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:48.004463 5975 server_base.cc:1034] running on GCE node
I20260810 03:50:48.004945 5975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:48.005687 5975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:48.006881 5975 hybrid_clock.cc:648] HybridClock initialized: now 1786333848006814 us; error 86 us; skew 500 ppm
Aug 10 03:50:48 dist-test-slave-pww0 krb5kdc[5781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333848, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:48.010414 5975 init.cc:377] Logged in from keytab as kudu/127.27.153.65@KRBTEST.COM (short username kudu)
I20260810 03:50:48.011704 5975 webserver.cc:460] Webserver started at http://127.27.153.65:39277/ using document root <none> and password file <none>
I20260810 03:50:48.012356 5975 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:48.012442 5975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:48.012671 5975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:48.015039 5975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "36cf7bbcf7b143dcb3e10c99990e21c7"
format_stamp: "Formatted at 2026-08-10 03:50:48 on dist-test-slave-pww0"
server_key: "53cca5da5b536cae081998abde58cd72"
server_key_iv: "9306836d5e9037c64fdbb2460e39fe5a"
server_key_version: "encryptionkey@0"
I20260810 03:50:48.015818 5975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "36cf7bbcf7b143dcb3e10c99990e21c7"
format_stamp: "Formatted at 2026-08-10 03:50:48 on dist-test-slave-pww0"
server_key: "53cca5da5b536cae081998abde58cd72"
server_key_iv: "9306836d5e9037c64fdbb2460e39fe5a"
server_key_version: "encryptionkey@0"
I20260810 03:50:48.020191 5975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:50:48.023151 5999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:48.024665 5975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:48.024816 5975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "36cf7bbcf7b143dcb3e10c99990e21c7"
format_stamp: "Formatted at 2026-08-10 03:50:48 on dist-test-slave-pww0"
server_key: "53cca5da5b536cae081998abde58cd72"
server_key_iv: "9306836d5e9037c64fdbb2460e39fe5a"
server_key_version: "encryptionkey@0"
I20260810 03:50:48.024955 5975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:48.040884 5975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:48.044538 5975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:48.044750 5975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:48.045410 5975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:48.046319 5975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:48.046368 5975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:48.046425 5975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:48.046454 5975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:48.056707 5975 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:46633
I20260810 03:50:48.056773 6198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:46633 every 8 connection(s)
I20260810 03:50:48.057708 5975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:50:48.064409 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 5975
I20260810 03:50:48.064532 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:50:48.064949 28261 external_mini_cluster.cc:1428] Setting key 79e68ff0717946842233b281f472e758
Aug 10 03:50:48 dist-test-slave-pww0 krb5kdc[5781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333848, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:48.073411 5937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:48.059771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:33975 (local address 127.27.153.126:34961)
0810 03:50:48.059970 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:48.059975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:48.060918 (+ 943us) server_negotiation.cc:407] Connection header received
0810 03:50:48.062348 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:48.062351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:48.062418 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:48.062497 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:48.063972 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:48.064772 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:48.066303 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:48.066661 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:48.066769 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:48.069539 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:48.069543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:48.069545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:48.069574 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:48.071033 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:48.071708 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:48.071712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:48.071713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:48.071802 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:48.072171 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:48.072174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:48.072175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:48.072388 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:48.072481 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:48.073066 (+ 585us) server_negotiation.cc:299] Negotiation successful
0810 03:50:48.073223 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 03:50:48.074373 6201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:48.060174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:34961 (local address 127.27.153.65:33975)
0810 03:50:48.060743 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:48.060792 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:48.062106 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:48.062716 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:48.062724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:48.063135 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:48.063760 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:48.063780 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:48.064955 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:48.064958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:48.066166 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:48.066180 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:48.066827 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:48.066830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:48.067005 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:48.067706 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:48.067725 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:48.069323 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:48.071187 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:48.071195 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:48.071216 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:48.071586 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:48.071894 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:48.071897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:48.071900 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:48.072062 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:48.072498 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:48.072507 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:48.072803 (+ 296us) client_negotiation.cc:769] Sending connection context
0810 03:50:48.072985 (+ 182us) client_negotiation.cc:240] Negotiation successful
0810 03:50:48.073388 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":121,"threads_started":1}
WARNING: no policy specified for kudu/127.27.153.66@KRBTEST.COM; defaulting to no policy
I20260810 03:50:48.075758 6199 heartbeater.cc:347] Connected to a master server at 127.27.153.126:34961
I20260810 03:50:48.076042 6199 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:48.076664 6199 heartbeater.cc:511] Master 127.27.153.126:34961 requested a full tablet report, sending...
I20260810 03:50:48.078564 5861 ts_manager.cc:194] Registered new tserver with Master: 36cf7bbcf7b143dcb3e10c99990e21c7 (127.27.153.65:46633)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.66@KRBTEST.COM" created.
I20260810 03:50:48.079936 5861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:33975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
I20260810 03:50:48.108208 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:34961
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:48.235898 6208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:48.236212 6208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:48.236258 6208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:48.240788 6208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:48.240892 6208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:48.240958 6208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:50:48.246712 6208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:34961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:48.248136 6208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:48.249346 6208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:48.261030 6215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:48.261049 6216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:48.261225 6208 server_base.cc:1034] running on GCE node
W20260810 03:50:48.261030 6221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:48.261893 6208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:48.262619 6208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:48.263804 6208 hybrid_clock.cc:648] HybridClock initialized: now 1786333848263765 us; error 54 us; skew 500 ppm
Aug 10 03:50:48 dist-test-slave-pww0 krb5kdc[5781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333848, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:48.267220 6208 init.cc:377] Logged in from keytab as kudu/127.27.153.66@KRBTEST.COM (short username kudu)
I20260810 03:50:48.268347 6208 webserver.cc:460] Webserver started at http://127.27.153.66:38209/ using document root <none> and password file <none>
I20260810 03:50:48.268945 6208 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:48.269021 6208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:48.269250 6208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:48.271795 6208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "f7644ad2fc6c45fe8456a546af6d9f03"
format_stamp: "Formatted at 2026-08-10 03:50:48 on dist-test-slave-pww0"
server_key: "b0504afc3875c032488c7fb779f1a227"
server_key_iv: "def55eabe2c22ad76978c5a5431cad3e"
server_key_version: "encryptionkey@0"
I20260810 03:50:48.272423 6208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "f7644ad2fc6c45fe8456a546af6d9f03"
format_stamp: "Formatted at 2026-08-10 03:50:48 on dist-test-slave-pww0"
server_key: "b0504afc3875c032488c7fb779f1a227"
server_key_iv: "def55eabe2c22ad76978c5a5431cad3e"
server_key_version: "encryptionkey@0"
I20260810 03:50:48.276661 6208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:50:48.279659 6231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:48.281301 6208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:48.281456 6208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "f7644ad2fc6c45fe8456a546af6d9f03"
format_stamp: "Formatted at 2026-08-10 03:50:48 on dist-test-slave-pww0"
server_key: "b0504afc3875c032488c7fb779f1a227"
server_key_iv: "def55eabe2c22ad76978c5a5431cad3e"
server_key_version: "encryptionkey@0"
I20260810 03:50:48.281596 6208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:48.293936 6208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:48.297957 6208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:48.298178 6208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:48.298877 6208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:48.299860 6208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:48.299911 6208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:48.299971 6208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:48.299999 6208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:48.309773 6208 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:44823
I20260810 03:50:48.309798 6420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:44823 every 8 connection(s)
I20260810 03:50:48.310786 6208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:50:48.317512 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 6208
I20260810 03:50:48.317634 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:50:48.317972 28261 external_mini_cluster.cc:1428] Setting key 9a7a60d6125fea1862a6559d53db880d
Aug 10 03:50:48 dist-test-slave-pww0 krb5kdc[5781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333848, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:48.326900 5937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:48.312821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:36183 (local address 127.27.153.126:34961)
0810 03:50:48.312952 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:48.312955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:48.313963 (+ 1008us) server_negotiation.cc:407] Connection header received
0810 03:50:48.315438 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:48.315442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:48.315548 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:48.315623 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:48.316996 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:48.317534 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:48.319165 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:48.319528 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:48.319641 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:48.322577 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:48.322581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:48.322584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:48.322618 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:48.324338 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:48.324962 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:48.324967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:48.324968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:48.325078 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:48.325463 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:48.325466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:48.325468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:48.325724 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:48.325828 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:48.326535 (+ 707us) server_negotiation.cc:299] Negotiation successful
0810 03:50:48.326700 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 03:50:48.327904 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:48.313186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:34961 (local address 127.27.153.66:36183)
0810 03:50:48.313767 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:48.313824 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:48.315194 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:48.315841 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:48.315848 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:48.316276 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:48.316819 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:48.316836 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:48.317721 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:48.317725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:48.319033 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:48.319046 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:48.319838 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:48.319841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:48.320017 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:48.320632 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:48.320650 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:48.322328 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:48.324508 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:48.324515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:48.324533 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:48.324842 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:48.325181 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:48.325183 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:48.325185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:48.325366 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:48.325872 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:48.325879 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:48.326198 (+ 319us) client_negotiation.cc:769] Sending connection context
0810 03:50:48.326470 (+ 272us) client_negotiation.cc:240] Negotiation successful
0810 03:50:48.326834 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":137,"threads_started":1}
I20260810 03:50:48.329408 6421 heartbeater.cc:347] Connected to a master server at 127.27.153.126:34961
I20260810 03:50:48.329732 6421 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.153.67@KRBTEST.COM; defaulting to no policy
I20260810 03:50:48.330348 6421 heartbeater.cc:511] Master 127.27.153.126:34961 requested a full tablet report, sending...
I20260810 03:50:48.331789 5861 ts_manager.cc:194] Registered new tserver with Master: f7644ad2fc6c45fe8456a546af6d9f03 (127.27.153.66:44823)
I20260810 03:50:48.332496 5861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:36183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
I20260810 03:50:48.350919 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:34961
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:48.474531 6428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:48.474845 6428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:48.474895 6428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:48.479377 6428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:48.479501 6428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:48.479583 6428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:50:48.486090 6428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:34961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:48.487357 6428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:48.488579 6428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:48.499945 6436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:48.499958 6435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:48.499979 6438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:48.500702 6428 server_base.cc:1034] running on GCE node
I20260810 03:50:48.501221 6428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:48.501991 6428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:48.503189 6428 hybrid_clock.cc:648] HybridClock initialized: now 1786333848503162 us; error 40 us; skew 500 ppm
Aug 10 03:50:48 dist-test-slave-pww0 krb5kdc[5781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333848, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:48.506629 6428 init.cc:377] Logged in from keytab as kudu/127.27.153.67@KRBTEST.COM (short username kudu)
I20260810 03:50:48.507867 6428 webserver.cc:460] Webserver started at http://127.27.153.67:45209/ using document root <none> and password file <none>
I20260810 03:50:48.508582 6428 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:48.508713 6428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:48.509056 6428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:48.511735 6428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "5f1fc507a3d843838bd7f516a106d7fe"
format_stamp: "Formatted at 2026-08-10 03:50:48 on dist-test-slave-pww0"
server_key: "17d369d0774a199e74e154c12a58a6de"
server_key_iv: "be1c69c6d2d15e5e5d6a482f567b9d1b"
server_key_version: "encryptionkey@0"
I20260810 03:50:48.512413 6428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "5f1fc507a3d843838bd7f516a106d7fe"
format_stamp: "Formatted at 2026-08-10 03:50:48 on dist-test-slave-pww0"
server_key: "17d369d0774a199e74e154c12a58a6de"
server_key_iv: "be1c69c6d2d15e5e5d6a482f567b9d1b"
server_key_version: "encryptionkey@0"
I20260810 03:50:48.517295 6428 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:50:48.520884 6450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:48.522943 6428 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:50:48.523103 6428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "5f1fc507a3d843838bd7f516a106d7fe"
format_stamp: "Formatted at 2026-08-10 03:50:48 on dist-test-slave-pww0"
server_key: "17d369d0774a199e74e154c12a58a6de"
server_key_iv: "be1c69c6d2d15e5e5d6a482f567b9d1b"
server_key_version: "encryptionkey@0"
I20260810 03:50:48.523264 6428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:48.543886 6428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:48.547701 6428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:48.547906 6428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:48.548588 6428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:48.549543 6428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:48.549595 6428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:48.549654 6428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:48.549685 6428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:48.559994 6428 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:40075
I20260810 03:50:48.560019 6633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:40075 every 8 connection(s)
I20260810 03:50:48.560992 6428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:50:48.570111 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 6428
I20260810 03:50:48.570261 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:50:48.570581 28261 external_mini_cluster.cc:1428] Setting key 3df943fa5d6033b45ecb7eeb00728cf4
Aug 10 03:50:48 dist-test-slave-pww0 krb5kdc[5781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333848, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:50:48.575968 5937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:48.563081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:33325 (local address 127.27.153.126:34961)
0810 03:50:48.563256 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:48.563261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:48.564248 (+ 987us) server_negotiation.cc:407] Connection header received
0810 03:50:48.565898 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:48.565902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:48.565967 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:48.566045 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:48.567356 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:48.567958 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:48.569154 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:48.569383 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:48.569479 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:48.572323 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:48.572326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:48.572328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:48.572356 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:48.573723 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:48.574291 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:48.574294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:48.574295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:48.574380 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:48.574727 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:48.574750 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:48.574751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:48.574955 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:48.575043 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:48.575656 (+ 613us) server_negotiation.cc:299] Negotiation successful
0810 03:50:48.575799 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 03:50:48.576880 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:48.563500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:34961 (local address 127.27.153.67:33325)
0810 03:50:48.564044 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:48.564113 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:48.565646 (+ 1533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:48.566252 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:48.566260 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:48.566661 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:48.567191 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:48.567208 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:48.568099 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:48.568102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:48.569012 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:48.569023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:48.569541 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:48.569545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:48.569717 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:48.570444 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:48.570462 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:48.572147 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:48.573849 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:48.573856 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:48.573876 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:48.574179 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:48.574468 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:48.574471 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:48.574473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:48.574634 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:48.575074 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:48.575082 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:48.575359 (+ 277us) client_negotiation.cc:769] Sending connection context
0810 03:50:48.575597 (+ 238us) client_negotiation.cc:240] Negotiation successful
0810 03:50:48.575970 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":140,"threads_started":1}
I20260810 03:50:48.578184 6634 heartbeater.cc:347] Connected to a master server at 127.27.153.126:34961
I20260810 03:50:48.578437 6634 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:48.578967 6634 heartbeater.cc:511] Master 127.27.153.126:34961 requested a full tablet report, sending...
I20260810 03:50:48.580122 5861 ts_manager.cc:194] Registered new tserver with Master: 5f1fc507a3d843838bd7f516a106d7fe (127.27.153.67:40075)
I20260810 03:50:48.580749 5861 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:33325
I20260810 03:50:48.585131 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/test-admin.keytab.
Aug 10 03:50:48 dist-test-slave-pww0 krb5kdc[5781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:48.605675 28261 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 03:50:48.608812 6643 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:50:48.607352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37678 (local address 127.27.153.65:46633)
0810 03:50:48.607703 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:48.607710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:48.607736 (+ 26us) server_negotiation.cc:407] Connection header received
0810 03:50:48.607792 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:48.607797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:48.607986 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:48.608113 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:48.608591 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37678: BlockingRecv error: recv got EOF from 127.0.0.1:37678 (error 108)
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":136,"threads_started":1}
W20260810 03:50:48.608932 28261 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/krb5cc' of type FILE
Aug 10 03:50:48 dist-test-slave-pww0 krb5kdc[5781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:48.610047 28261 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 03:50:48 dist-test-slave-pww0 krb5kdc[5781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:48.611105 28261 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 03:50:48 dist-test-slave-pww0 krb5kdc[5781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.65@KRBTEST.COM
I20260810 03:50:48.621846 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:48.612162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37690 (local address 127.27.153.65:46633)
0810 03:50:48.612316 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:48.612321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:48.612406 (+ 85us) server_negotiation.cc:407] Connection header received
0810 03:50:48.612501 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:48.612504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:48.612552 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:48.612624 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:48.613225 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:48.614060 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:48.614989 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:48.615410 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:48.615579 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:48.618187 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:48.618201 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:48.618219 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:48.618254 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:48.620021 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:48.620524 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:48.620529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:48.620530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:48.620610 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:48.620837 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:48.620839 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:48.620841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:48.621107 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:48.621287 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:48.621476 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 03:50:48.621647 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260810 03:50:48.626431 6643 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:50:48.625356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37700 (local address 127.27.153.65:46633)
0810 03:50:48.625480 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:48.625484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:48.625563 (+ 79us) server_negotiation.cc:407] Connection header received
0810 03:50:48.625730 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:48.625733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:48.625776 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:48.625840 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:48.626309 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37700: BlockingRecv error: recv got EOF from 127.0.0.1:37700 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 03:50:48.626586 28261 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/krb5cc' of type FILE
Aug 10 03:50:48 dist-test-slave-pww0 krb5kdc[5781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:48.627769 28261 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 03:50:48 dist-test-slave-pww0 krb5kdc[5781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:48.628875 28261 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 03:50:48 dist-test-slave-pww0 krb5kdc[5781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.65@KRBTEST.COM
I20260810 03:50:48.638454 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:48.629993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37702 (local address 127.27.153.65:46633)
0810 03:50:48.630120 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:48.630124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:48.630239 (+ 115us) server_negotiation.cc:407] Connection header received
0810 03:50:48.630390 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:48.630393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:48.630436 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:48.630497 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:48.631043 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:48.631633 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:48.632440 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:48.632686 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:48.632775 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:48.635181 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:48.635185 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:48.635187 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:48.635217 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:48.636669 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:48.637148 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:48.637153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:48.637156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:48.637267 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:48.637569 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:48.637571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:48.637572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:48.637764 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:48.637893 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:48.638208 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 03:50:48.638329 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 03:50:48.641520 6643 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:50:48.640504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37716 (local address 127.27.153.65:46633)
0810 03:50:48.640624 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:48.640628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:48.640760 (+ 132us) server_negotiation.cc:407] Connection header received
0810 03:50:48.640884 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:48.640887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:48.640928 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:48.640990 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:48.641414 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37716: BlockingRecv error: recv got EOF from 127.0.0.1:37716 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 03:50:48.641793 28261 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786333814837068-28261-0/krb5kdc/krb5cc' of type FILE
Aug 10 03:50:48 dist-test-slave-pww0 krb5kdc[5781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:48.642874 28261 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 03:50:48 dist-test-slave-pww0 krb5kdc[5781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:48.643971 28261 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 03:50:48 dist-test-slave-pww0 krb5kdc[5781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.65@KRBTEST.COM
I20260810 03:50:48.653246 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:48.644947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37726 (local address 127.27.153.65:46633)
0810 03:50:48.645077 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:48.645082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:48.645220 (+ 138us) server_negotiation.cc:407] Connection header received
0810 03:50:48.645304 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:48.645306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:48.645349 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:48.645418 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:48.645936 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:48.646480 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:48.647335 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:48.647632 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:48.647730 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:48.650031 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:48.650035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:48.650037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:48.650067 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:48.651410 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:48.652092 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:48.652095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:48.652096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:48.652189 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:48.652499 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:48.652501 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:48.652502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:48.652685 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:48.652830 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:48.653013 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 03:50:48.653121 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 03:50:48.654506 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 5975
I20260810 03:50:48.668725 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 6208
I20260810 03:50:48.683794 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 6428
I20260810 03:50:48.715870 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 5809
2026-08-10T03:50:48Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2551 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:50:48 dist-test-slave-pww0 krb5kdc[6667](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:50:48 dist-test-slave-pww0 krb5kdc[6667](info): set up 2 sockets
Aug 10 03:50:48 dist-test-slave-pww0 krb5kdc[6667](info): commencing operation
krb5kdc: starting...
W20260810 03:50:49.940572 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.195s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:50:50 dist-test-slave-pww0 krb5kdc[6667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:50:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:50:50Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.126.keytab.
Entry for principal oryx/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.126.keytab.
I20260810 03:50:50.078334 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:41933
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.126.keytab
--principal=oryx/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:41933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:50.197356 6697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:50.197696 6697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:50.197757 6697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:50.202445 6697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:50:50.202543 6697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:50.202569 6697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:50.202591 6697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:50:50.202633 6697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:50:50.208580 6697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:41933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.126.keytab
--principal=oryx/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:41933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:50.209818 6697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:50.211025 6697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:50.220391 6708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:50.220407 6714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:50.220481 6697 server_base.cc:1034] running on GCE node
W20260810 03:50:50.220391 6707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:50.221356 6697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:50.222640 6697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:50.223858 6697 hybrid_clock.cc:648] HybridClock initialized: now 1786333850223838 us; error 37 us; skew 500 ppm
Aug 10 03:50:50 dist-test-slave-pww0 krb5kdc[6667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333850, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:50.227200 6697 init.cc:377] Logged in from keytab as oryx/127.27.153.126@KRBTEST.COM (short username oryx)
I20260810 03:50:50.228458 6697 webserver.cc:460] Webserver started at http://127.27.153.126:44401/ using document root <none> and password file <none>
I20260810 03:50:50.229060 6697 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:50.229137 6697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:50.229354 6697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:50.231773 6697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b"
format_stamp: "Formatted at 2026-08-10 03:50:50 on dist-test-slave-pww0"
server_key: "817e3b3225aa3481438bd368e6d73d7b"
server_key_iv: "83617f3e36a90d6f2ae92a5b7b9cc31c"
server_key_version: "encryptionkey@0"
I20260810 03:50:50.232410 6697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b"
format_stamp: "Formatted at 2026-08-10 03:50:50 on dist-test-slave-pww0"
server_key: "817e3b3225aa3481438bd368e6d73d7b"
server_key_iv: "83617f3e36a90d6f2ae92a5b7b9cc31c"
server_key_version: "encryptionkey@0"
I20260810 03:50:50.236554 6697 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:50:50.239233 6728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:50.240592 6697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:50.240756 6697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b"
format_stamp: "Formatted at 2026-08-10 03:50:50 on dist-test-slave-pww0"
server_key: "817e3b3225aa3481438bd368e6d73d7b"
server_key_iv: "83617f3e36a90d6f2ae92a5b7b9cc31c"
server_key_version: "encryptionkey@0"
I20260810 03:50:50.240871 6697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:50.281358 6697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:50.284777 6697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:50.284977 6697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:50.292536 6697 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:41933
I20260810 03:50:50.292812 6828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:41933 every 8 connection(s)
I20260810 03:50:50.293598 6697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:50:50.296684 6832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:50.296762 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 6697
I20260810 03:50:50.296895 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:50:50.297173 28261 external_mini_cluster.cc:1428] Setting key ab5411180f801eab69a1f942ccfd1751
I20260810 03:50:50.304206 6832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b: Bootstrap starting.
Aug 10 03:50:50 dist-test-slave-pww0 krb5kdc[6667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM
I20260810 03:50:50.307235 6832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:50.308225 6832 log.cc:824] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:50.310977 6832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b: No bootstrap required, opened a new log
I20260810 03:50:50.313131 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:50.298649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46396 (local address 127.27.153.126:41933)
0810 03:50:50.299108 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:50.299123 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:50.299161 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:50:50.300023 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:50.300045 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:50.300413 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:50.300717 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:50.301748 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:50.302806 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:50.304040 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:50.304580 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:50.304735 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:50.307704 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:50.307711 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:50.307730 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:50.307771 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:50.309759 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:50.310398 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:50.310404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:50.310409 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:50.310552 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:50.310934 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:50.310937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:50.310939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:50.311520 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:50.311679 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:50.311969 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 03:50:50.312295 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":122,"threads_started":1}
I20260810 03:50:50.314991 6832 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } }
I20260810 03:50:50.315253 6832 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:50.315348 6832 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3d2c2bbe823423ca1284d9fa2a52b2b, State: Initialized, Role: FOLLOWER
I20260810 03:50:50.315861 6832 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } }
I20260810 03:50:50.315997 6832 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:50:50.316046 6832 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:50:50.316145 6832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:50.317137 6832 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } }
I20260810 03:50:50.317466 6832 leader_election.cc:302] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3d2c2bbe823423ca1284d9fa2a52b2b; no voters:
I20260810 03:50:50.317795 6832 leader_election.cc:288] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:50:50.317955 6839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:50.318212 6839 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [term 1 LEADER]: Becoming Leader. State: Replica: f3d2c2bbe823423ca1284d9fa2a52b2b, State: Running, Role: LEADER
I20260810 03:50:50.318529 6839 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } }
I20260810 03:50:50.319061 6832 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:50:50.320276 6843 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3d2c2bbe823423ca1284d9fa2a52b2b. Latest consensus state: current_term: 1 leader_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } }
I20260810 03:50:50.320418 6843 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:50.320698 6840 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } }
I20260810 03:50:50.320792 6840 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [sys.catalog]: This master's current role is: LEADER
W20260810 03:50:50.324357 6869 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:50:50.324450 6869 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:50:50.324611 6859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:50:50.325598 6859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:50:50.335276 6859 catalog_manager.cc:1347] Generated new cluster ID: f35189ad36ec45d1a449e600cbd8824f
I20260810 03:50:50.335400 6859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:50:50.348446 6859 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:50:50.349858 6859 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:50:50.359519 6859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b: Generated new TSK 0
I20260810 03:50:50.360318 6859 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.153.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.153.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.153.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.65.keytab.
Entry for principal oryx/127.27.153.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.65.keytab.
I20260810 03:50:50.398119 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:41933
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.65.keytab
--principal=oryx/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:50.516784 6883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:50.517072 6883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:50.517138 6883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:50.521483 6883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:50.521565 6883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:50.521656 6883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:50:50.527341 6883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.65.keytab
--principal=oryx/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:41933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:50.528566 6883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:50.529716 6883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:50.540113 6896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:50.540129 6895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:50.540091 6900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:50.540452 6883 server_base.cc:1034] running on GCE node
I20260810 03:50:50.540938 6883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:50.541630 6883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:50.542795 6883 hybrid_clock.cc:648] HybridClock initialized: now 1786333850542772 us; error 34 us; skew 500 ppm
Aug 10 03:50:50 dist-test-slave-pww0 krb5kdc[6667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333850, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:50.545805 6883 init.cc:377] Logged in from keytab as oryx/127.27.153.65@KRBTEST.COM (short username oryx)
I20260810 03:50:50.546805 6883 webserver.cc:460] Webserver started at http://127.27.153.65:37765/ using document root <none> and password file <none>
I20260810 03:50:50.547358 6883 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:50.547418 6883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:50.547640 6883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:50.549778 6883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "6720a63653bd424bba05965f0b0844dc"
format_stamp: "Formatted at 2026-08-10 03:50:50 on dist-test-slave-pww0"
server_key: "1e241a55e328c88fd30358594b68e330"
server_key_iv: "bd625c001f92dd96185b1d007db4ef3a"
server_key_version: "encryptionkey@0"
I20260810 03:50:50.550331 6883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "6720a63653bd424bba05965f0b0844dc"
format_stamp: "Formatted at 2026-08-10 03:50:50 on dist-test-slave-pww0"
server_key: "1e241a55e328c88fd30358594b68e330"
server_key_iv: "bd625c001f92dd96185b1d007db4ef3a"
server_key_version: "encryptionkey@0"
I20260810 03:50:50.554284 6883 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:50:50.557103 6914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:50.558609 6883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:50:50.558782 6883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "6720a63653bd424bba05965f0b0844dc"
format_stamp: "Formatted at 2026-08-10 03:50:50 on dist-test-slave-pww0"
server_key: "1e241a55e328c88fd30358594b68e330"
server_key_iv: "bd625c001f92dd96185b1d007db4ef3a"
server_key_version: "encryptionkey@0"
I20260810 03:50:50.558917 6883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:50.579063 6883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:50.582459 6883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:50.582659 6883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:50.583302 6883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:50.584231 6883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:50.584281 6883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:50.584340 6883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:50.584370 6883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:50.593804 6883 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:35535
I20260810 03:50:50.593894 7116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:35535 every 8 connection(s)
I20260810 03:50:50.594969 6883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:50:50.596424 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 6883
I20260810 03:50:50.596534 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:50:50.596805 28261 external_mini_cluster.cc:1428] Setting key 340e307fc902e2a5f92972736142c91a
Aug 10 03:50:50 dist-test-slave-pww0 krb5kdc[6667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333850, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.27.153.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.153.66@KRBTEST.COM" created.
I20260810 03:50:50.610459 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:50.597034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:45643 (local address 127.27.153.126:41933)
0810 03:50:50.597188 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:50.597194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:50.598228 (+ 1034us) server_negotiation.cc:407] Connection header received
0810 03:50:50.599752 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:50.599757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:50.599814 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:50.599890 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:50.601193 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:50.601795 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:50.602981 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:50.603246 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:50.603341 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:50.606264 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:50.606268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:50.606270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:50.606300 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:50.607751 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:50.608357 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:50.608361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:50.608362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:50.608456 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:50.608938 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:50.608941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:50.608942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:50.609195 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:50.609290 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:50.610094 (+ 804us) server_negotiation.cc:299] Negotiation successful
0810 03:50:50.610278 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:50:50.611251 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:50.597521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.65:45643)
0810 03:50:50.598072 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:50.598127 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:50.599526 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:50.600083 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:50.600090 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:50.600494 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:50.601041 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:50.601056 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:50.601927 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:50.601930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:50.602853 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:50.602864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:50.603412 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:50.603416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:50.603653 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:50.604262 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:50.604280 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:50.606030 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:50.607897 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:50.607902 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:50.607919 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:50.608223 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:50.608592 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:50.608595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:50.608596 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:50.608792 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:50.609350 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:50.609356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:50.609765 (+ 409us) client_negotiation.cc:769] Sending connection context
0810 03:50:50.609987 (+ 222us) client_negotiation.cc:240] Negotiation successful
0810 03:50:50.610310 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":128,"threads_started":1}
I20260810 03:50:50.612804 7117 heartbeater.cc:347] Connected to a master server at 127.27.153.126:41933
I20260810 03:50:50.613109 7117 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:50.613782 7117 heartbeater.cc:511] Master 127.27.153.126:41933 requested a full tablet report, sending...
I20260810 03:50:50.615859 6760 ts_manager.cc:194] Registered new tserver with Master: 6720a63653bd424bba05965f0b0844dc (127.27.153.65:35535)
I20260810 03:50:50.617563 6760 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:45643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.153.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.66.keytab.
Entry for principal oryx/127.27.153.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.66.keytab.
I20260810 03:50:50.630702 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:41933
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.66.keytab
--principal=oryx/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:50.749585 7127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:50.749894 7127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:50.749961 7127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:50.754134 7127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:50.754215 7127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:50.754303 7127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:50:50.760452 7127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.66.keytab
--principal=oryx/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:41933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:50.761723 7127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:50.762899 7127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:50.773160 7137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:50.773132 7136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:50.773231 7127 server_base.cc:1034] running on GCE node
W20260810 03:50:50.773132 7140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:50.773867 7127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:50.774513 7127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:50.775672 7127 hybrid_clock.cc:648] HybridClock initialized: now 1786333850775651 us; error 36 us; skew 500 ppm
Aug 10 03:50:50 dist-test-slave-pww0 krb5kdc[6667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333850, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:50.778527 7127 init.cc:377] Logged in from keytab as oryx/127.27.153.66@KRBTEST.COM (short username oryx)
I20260810 03:50:50.779556 7127 webserver.cc:460] Webserver started at http://127.27.153.66:46207/ using document root <none> and password file <none>
I20260810 03:50:50.780134 7127 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:50.780198 7127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:50.780402 7127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:50.782624 7127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "7a083bc0c0ba48e79563564743d2fb3f"
format_stamp: "Formatted at 2026-08-10 03:50:50 on dist-test-slave-pww0"
server_key: "2e43319a214928dd921d54318779336f"
server_key_iv: "8594e276226afea000f6b8ad9092f3e5"
server_key_version: "encryptionkey@0"
I20260810 03:50:50.783205 7127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "7a083bc0c0ba48e79563564743d2fb3f"
format_stamp: "Formatted at 2026-08-10 03:50:50 on dist-test-slave-pww0"
server_key: "2e43319a214928dd921d54318779336f"
server_key_iv: "8594e276226afea000f6b8ad9092f3e5"
server_key_version: "encryptionkey@0"
I20260810 03:50:50.787586 7127 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 03:50:50.790436 7155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:50.792130 7127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:50.792284 7127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "7a083bc0c0ba48e79563564743d2fb3f"
format_stamp: "Formatted at 2026-08-10 03:50:50 on dist-test-slave-pww0"
server_key: "2e43319a214928dd921d54318779336f"
server_key_iv: "8594e276226afea000f6b8ad9092f3e5"
server_key_version: "encryptionkey@0"
I20260810 03:50:50.792425 7127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:50.817240 7127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:50.822258 7127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:50.822502 7127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:50.823158 7127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:50.824105 7127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:50.824157 7127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:50.824209 7127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:50.824239 7127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:50.833652 7127 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:41001
I20260810 03:50:50.833698 7355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:41001 every 8 connection(s)
I20260810 03:50:50.834628 7127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:50:50.839406 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 7127
I20260810 03:50:50.839591 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:50:50.839895 28261 external_mini_cluster.cc:1428] Setting key 04691bb00b6302f7b8377e1bad531945
Aug 10 03:50:50 dist-test-slave-pww0 krb5kdc[6667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333850, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM
I20260810 03:50:50.851334 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:50.836660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:54995 (local address 127.27.153.126:41933)
0810 03:50:50.836819 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:50.836823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:50.837748 (+ 925us) server_negotiation.cc:407] Connection header received
0810 03:50:50.839192 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:50.839196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:50.839265 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:50.839391 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:50.841060 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:50.841679 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:50.842974 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:50.843291 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:50.843422 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:50.846811 (+ 3389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:50.846817 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:50.846820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:50.846858 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:50.848669 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:50.849311 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:50.849316 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:50.849318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:50.849431 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:50.849885 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:50.849890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:50.849892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:50.850196 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:50.850332 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:50.850963 (+ 631us) server_negotiation.cc:299] Negotiation successful
0810 03:50:50.851128 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":63}
WARNING: no policy specified for oryx/127.27.153.67@KRBTEST.COM; defaulting to no policy
I20260810 03:50:50.852190 7367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:50.837044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.66:54995)
0810 03:50:50.837562 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:50.837612 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:50.838921 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:50.839727 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:50.839736 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:50.840188 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:50.840867 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:50.840890 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:50.841840 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:50.841844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:50.842822 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:50.842833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:50.843564 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:50.843569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:50.843844 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:50.844689 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:50.844714 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:50.846567 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:50.848850 (+ 2283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:50.848857 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:50.848873 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:50.849189 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:50.849569 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:50.849572 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:50.849574 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:50.849770 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:50.850354 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:50.850361 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:50.850675 (+ 314us) client_negotiation.cc:769] Sending connection context
0810 03:50:50.850862 (+ 187us) client_negotiation.cc:240] Negotiation successful
0810 03:50:50.851185 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":138,"threads_started":1}
I20260810 03:50:50.853686 7364 heartbeater.cc:347] Connected to a master server at 127.27.153.126:41933
I20260810 03:50:50.853973 7364 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:50.854671 7364 heartbeater.cc:511] Master 127.27.153.126:41933 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.153.67@KRBTEST.COM" created.
I20260810 03:50:50.856005 6760 ts_manager.cc:194] Registered new tserver with Master: 7a083bc0c0ba48e79563564743d2fb3f (127.27.153.66:41001)
I20260810 03:50:50.856873 6760 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:54995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.153.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.67.keytab.
Entry for principal oryx/127.27.153.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.67.keytab.
I20260810 03:50:50.874914 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:41933
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.67.keytab
--principal=oryx/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:50.999526 7371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:50.999841 7371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:50.999907 7371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:51.004552 7371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:51.004637 7371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:51.004727 7371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:50:51.010667 7371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.67.keytab
--principal=oryx/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:41933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:51.011907 7371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:51.013103 7371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:51.023635 7385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:51.023639 7384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:51.023639 7390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:51.024219 7371 server_base.cc:1034] running on GCE node
I20260810 03:50:51.024722 7371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:51.025553 7371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:51.026743 7371 hybrid_clock.cc:648] HybridClock initialized: now 1786333851026711 us; error 56 us; skew 500 ppm
Aug 10 03:50:51 dist-test-slave-pww0 krb5kdc[6667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333851, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:51.030277 7371 init.cc:377] Logged in from keytab as oryx/127.27.153.67@KRBTEST.COM (short username oryx)
I20260810 03:50:51.031359 7371 webserver.cc:460] Webserver started at http://127.27.153.67:38001/ using document root <none> and password file <none>
I20260810 03:50:51.031996 7371 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:51.032068 7371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:51.032292 7371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:51.035012 7371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "16dc897a7f744e78a9b0e59c3abd6d11"
format_stamp: "Formatted at 2026-08-10 03:50:51 on dist-test-slave-pww0"
server_key: "e27d02d96fd8c18dfe4499637c3e220d"
server_key_iv: "34cccd6bcf64359f57816673afa48805"
server_key_version: "encryptionkey@0"
I20260810 03:50:51.035677 7371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "16dc897a7f744e78a9b0e59c3abd6d11"
format_stamp: "Formatted at 2026-08-10 03:50:51 on dist-test-slave-pww0"
server_key: "e27d02d96fd8c18dfe4499637c3e220d"
server_key_iv: "34cccd6bcf64359f57816673afa48805"
server_key_version: "encryptionkey@0"
I20260810 03:50:51.040040 7371 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:50:51.042949 7403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:51.044620 7371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:51.044764 7371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "16dc897a7f744e78a9b0e59c3abd6d11"
format_stamp: "Formatted at 2026-08-10 03:50:51 on dist-test-slave-pww0"
server_key: "e27d02d96fd8c18dfe4499637c3e220d"
server_key_iv: "34cccd6bcf64359f57816673afa48805"
server_key_version: "encryptionkey@0"
I20260810 03:50:51.044883 7371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:51.068758 7371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:51.072573 7371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:51.072801 7371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:51.073487 7371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:51.074411 7371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:51.074463 7371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:51.074520 7371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:51.074549 7371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:51.083665 7371 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:45049
I20260810 03:50:51.083695 7606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:45049 every 8 connection(s)
I20260810 03:50:51.084668 7371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:50:51.094221 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 7371
I20260810 03:50:51.094373 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:50:51.094704 28261 external_mini_cluster.cc:1428] Setting key c85728f345f2eba7d46eb34956140827
Aug 10 03:50:51 dist-test-slave-pww0 krb5kdc[6667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333851, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM
I20260810 03:50:51.100229 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.086723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:42457 (local address 127.27.153.126:41933)
0810 03:50:51.086906 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:51.086910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:51.087870 (+ 960us) server_negotiation.cc:407] Connection header received
0810 03:50:51.089394 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:51.089398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:51.089465 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:51.089560 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:51.090925 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.091551 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.092773 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.093000 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.093097 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.096331 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:51.096334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:51.096336 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:51.096364 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:51.097844 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.098422 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.098427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.098429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.098566 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.098912 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.098914 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.098916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.099174 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:51.099266 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:51.099959 (+ 693us) server_negotiation.cc:299] Negotiation successful
0810 03:50:51.100084 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 03:50:51.101133 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.087138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.67:42457)
0810 03:50:51.087711 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:51.087760 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:51.089079 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:51.089771 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:51.089779 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:51.090176 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:51.090751 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.090768 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.091687 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.091690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.092623 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.092646 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.093164 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.093169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.093360 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.094055 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:51.094078 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:51.096130 (+ 2052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:51.097956 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.097961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.097976 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.098281 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.098657 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.098659 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.098660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.098834 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.099313 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:51.099319 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:51.099700 (+ 381us) client_negotiation.cc:769] Sending connection context
0810 03:50:51.099889 (+ 189us) client_negotiation.cc:240] Negotiation successful
0810 03:50:51.100192 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":131,"threads_started":1}
I20260810 03:50:51.102371 7607 heartbeater.cc:347] Connected to a master server at 127.27.153.126:41933
I20260810 03:50:51.102640 7607 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:51.103191 7607 heartbeater.cc:511] Master 127.27.153.126:41933 requested a full tablet report, sending...
I20260810 03:50:51.104427 6760 ts_manager.cc:194] Registered new tserver with Master: 16dc897a7f744e78a9b0e59c3abd6d11 (127.27.153.67:45049)
I20260810 03:50:51.105103 6760 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:42457
I20260810 03:50:51.109426 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.153.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.153.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.153.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.125.keytab.
Entry for principal oryx/127.27.153.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.125.keytab.
I20260810 03:50:51.140682 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.125:45527
--webserver_interface=127.27.153.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.125.keytab
--principal=oryx/127.27.153.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:41933,127.27.153.125:45527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:51.261044 7619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:51.261343 7619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:51.261384 7619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:51.265743 7619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:50:51.265826 7619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:51.265846 7619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:51.265862 7619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:50:51.265877 7619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:50:51.272047 7619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:41933,127.27.153.125:45527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.125.keytab
--principal=oryx/127.27.153.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.125:45527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.153.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
W20260810 03:50:51.272320 7619 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.153.126:41933,127.27.153.125:45527'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 03:50:51.273272 7619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:51.274308 7619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:51.283728 7632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:51.283756 7619 server_base.cc:1034] running on GCE node
W20260810 03:50:51.283756 7636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:51.283891 7631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:51.284653 7619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:51.285867 7619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:51.287063 7619 hybrid_clock.cc:648] HybridClock initialized: now 1786333851287041 us; error 34 us; skew 500 ppm
Aug 10 03:50:51 dist-test-slave-pww0 krb5kdc[6667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333851, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:51.294718 7619 init.cc:377] Logged in from keytab as oryx/127.27.153.125@KRBTEST.COM (short username oryx)
I20260810 03:50:51.295925 7619 webserver.cc:460] Webserver started at http://127.27.153.125:36003/ using document root <none> and password file <none>
I20260810 03:50:51.296559 7619 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:51.296643 7619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:51.296877 7619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:51.299182 7619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/data/instance:
uuid: "5e655481b21e4dd4ac7c1135dbb4d855"
format_stamp: "Formatted at 2026-08-10 03:50:51 on dist-test-slave-pww0"
server_key: "f64ae5335180e1d30fd1661ebe886301"
server_key_iv: "619e951d6d4e27c0d5a8e7f7c0b86e05"
server_key_version: "encryptionkey@0"
I20260810 03:50:51.299808 7619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/wal/instance:
uuid: "5e655481b21e4dd4ac7c1135dbb4d855"
format_stamp: "Formatted at 2026-08-10 03:50:51 on dist-test-slave-pww0"
server_key: "f64ae5335180e1d30fd1661ebe886301"
server_key_iv: "619e951d6d4e27c0d5a8e7f7c0b86e05"
server_key_version: "encryptionkey@0"
I20260810 03:50:51.303910 7619 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:50:51.306777 7652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:51.308444 7619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:51.308599 7619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/wal
uuid: "5e655481b21e4dd4ac7c1135dbb4d855"
format_stamp: "Formatted at 2026-08-10 03:50:51 on dist-test-slave-pww0"
server_key: "f64ae5335180e1d30fd1661ebe886301"
server_key_iv: "619e951d6d4e27c0d5a8e7f7c0b86e05"
server_key_version: "encryptionkey@0"
I20260810 03:50:51.308722 7619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:51.331360 7619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:51.334919 7619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:51.335134 7619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:51.342521 7619 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.125:45527
I20260810 03:50:51.342836 7740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.125:45527 every 8 connection(s)
I20260810 03:50:51.343640 7619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/data/info.pb
I20260810 03:50:51.347067 7742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:51.349051 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 7619
I20260810 03:50:51.352015 7742 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:50:51 dist-test-slave-pww0 krb5kdc[6667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.153.125@KRBTEST.COM
Aug 10 03:50:51 dist-test-slave-pww0 krb5kdc[6667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333851, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.125@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM
I20260810 03:50:51.364265 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.354402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46412 (local address 127.27.153.126:41933)
0810 03:50:51.354546 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:51.354551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:51.354906 (+ 355us) server_negotiation.cc:407] Connection header received
0810 03:50:51.355037 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:51.355039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:51.355083 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:51.355157 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:51.355981 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.356504 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.357422 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.357635 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.357722 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.360678 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:51.360683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:51.360686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:51.360717 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:51.362251 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.362754 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.362758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.362759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.362872 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.363183 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.363186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.363188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.363435 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:51.363557 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:51.363963 (+ 406us) server_negotiation.cc:299] Negotiation successful
0810 03:50:51.364097 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:50:51.365254 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.350960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39100 (local address 127.27.153.125:45527)
0810 03:50:51.351499 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:51.351521 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:51.351569 (+ 48us) server_negotiation.cc:407] Connection header received
0810 03:50:51.352389 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:51.352417 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:51.352914 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:51.353317 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:51.354046 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.355482 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.356408 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.356926 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.357078 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.360054 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:51.360063 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:51.360079 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:51.360108 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:51.362241 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.362749 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.362754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.362756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.362932 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.363223 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.363226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.363228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.363719 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:51.363924 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:51.364217 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 03:50:51.364654 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":260,"server-negotiator.queue_time_us":340,"thread_start_us":141,"threads_started":1}
I20260810 03:50:51.365254 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.354348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.0.0.1:46412)
0810 03:50:51.354768 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:51.354823 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:51.354953 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:51.355220 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:51.355226 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:51.355710 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:51.355850 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.355856 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.356630 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.356633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.357296 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.357306 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.357790 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.357794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.357884 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.358456 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:51.358474 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:51.360476 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:51.362374 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.362384 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.362397 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.362644 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.362962 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.362966 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.362968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.363095 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.363556 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:51.363563 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:51.363742 (+ 179us) client_negotiation.cc:769] Sending connection context
0810 03:50:51.363901 (+ 159us) client_negotiation.cc:240] Negotiation successful
0810 03:50:51.364350 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"spinlock_wait_cycles":1920,"thread_start_us":96,"threads_started":1}
I20260810 03:50:51.367489 7742 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:50:51 dist-test-slave-pww0 krb5kdc[6667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333851, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.125@KRBTEST.COM for oryx/127.27.153.125@KRBTEST.COM
I20260810 03:50:51.376888 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.367998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:45527 (local address 127.0.0.1:39104)
0810 03:50:51.368256 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:51.368268 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:51.368355 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:51.368566 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:51.368568 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:51.368721 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:51.368824 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.368829 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.370234 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.370236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.370790 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.370799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.371194 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.371196 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.371271 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.371758 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:51.371770 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:51.373004 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:51.375434 (+ 2430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.375437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.375438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.375671 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.375899 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.375901 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.375902 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.375976 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.376412 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:51.376415 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:51.376506 (+ 91us) client_negotiation.cc:769] Sending connection context
0810 03:50:51.376597 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 03:50:51.376753 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260810 03:50:51.376974 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.368137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39104 (local address 127.27.153.125:45527)
0810 03:50:51.368255 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:51.368258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:51.368370 (+ 112us) server_negotiation.cc:407] Connection header received
0810 03:50:51.368419 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:51.368421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:51.368464 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:51.368551 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:51.368941 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.369933 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.370884 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.371100 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.371172 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.373130 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:51.373133 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:51.373135 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:51.373158 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:51.375302 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.375752 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.375754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.375756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.375830 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.376076 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.376078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.376080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.376300 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:51.376394 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:51.376665 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 03:50:51.376825 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":42}
I20260810 03:50:51.379998 7742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855: Bootstrap starting.
I20260810 03:50:51.382405 7742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:51.383142 7742 log.cc:824] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:51.385349 7742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855: No bootstrap required, opened a new log
I20260810 03:50:51.388495 7742 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } }
I20260810 03:50:51.388728 7742 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:51.388773 7742 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e655481b21e4dd4ac7c1135dbb4d855, State: Initialized, Role: FOLLOWER
I20260810 03:50:51.389204 7742 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } }
I20260810 03:50:51.389765 7751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } } }
I20260810 03:50:51.389890 7751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:50:51.390542 7742 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 03:50:51.395515 7778 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:50:51.395608 7778 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:50:51.398233 7619 master_runner.cc:428] Detected that this master 5e655481b21e4dd4ac7c1135dbb4d855 is joining an existing cluster
I20260810 03:50:51.398298 7619 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.153.125:45527
I20260810 03:50:51.408949 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.400604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46414 (local address 127.27.153.126:41933)
0810 03:50:51.400721 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:51.400726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:51.401144 (+ 418us) server_negotiation.cc:407] Connection header received
0810 03:50:51.401342 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:51.401346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:51.401397 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:51.401475 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:51.402125 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.402911 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.403871 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.404109 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.404207 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.405648 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:51.405652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:51.405655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:51.405684 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:51.407117 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.407675 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.407679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.407681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.407764 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.408013 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.408016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.408018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.408232 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:51.408321 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:51.408647 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 03:50:51.408787 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 03:50:51.408946 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.400450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.0.0.1:46414)
0810 03:50:51.401051 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:51.401066 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:51.401232 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:51.401503 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:51.401506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:51.401836 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:51.402014 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.402023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.403028 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.403030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.403754 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.403764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.404251 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.404254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.404409 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.405019 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:51.405034 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:51.405494 (+ 460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:51.407232 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.407235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.407237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.407577 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.407845 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.407848 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.407850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.407931 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.408334 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:51.408337 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:51.408439 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 03:50:51.408564 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 03:50:51.408733 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":100,"spinlock_wait_cycles":3840,"thread_start_us":120,"threads_started":1}
I20260810 03:50:51.409602 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.400757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:45527 (local address 127.0.0.1:39114)
0810 03:50:51.401054 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:51.401065 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:51.401169 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:51.401479 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:51.401481 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:51.401806 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:51.401976 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.401983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.403004 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.403008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.403998 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.404010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.404551 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.404555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.404652 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.405394 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:51.405408 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:51.405787 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:51.407751 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.407754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.407756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.408004 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.408296 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.408299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.408300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.408374 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.408797 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:51.408800 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:51.408877 (+ 77us) client_negotiation.cc:769] Sending connection context
0810 03:50:51.409344 (+ 467us) client_negotiation.cc:240] Negotiation successful
0810 03:50:51.409459 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":18,"thread_start_us":96,"threads_started":1}
I20260810 03:50:51.409307 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.400471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.27.153.125:45527)
0810 03:50:51.400597 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:51.400601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:51.401151 (+ 550us) server_negotiation.cc:407] Connection header received
0810 03:50:51.401258 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:51.401261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:51.401302 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:51.401366 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:51.402088 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.402888 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.404140 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.404444 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.404531 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.405919 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:51.405923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:51.405926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:51.405950 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:51.407540 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.408100 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.408103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.408105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.408176 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.408456 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.408459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.408461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.408679 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:51.408764 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:51.409032 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 03:50:51.409168 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":39,"spinlock_wait_cycles":1280}
Aug 10 03:50:51 dist-test-slave-pww0 krb5kdc[6667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333850, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.126@KRBTEST.COM for oryx/127.27.153.125@KRBTEST.COM
I20260810 03:50:51.425580 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.415051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.27.153.125:45527)
0810 03:50:51.415187 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:51.415192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:51.415492 (+ 300us) server_negotiation.cc:407] Connection header received
0810 03:50:51.415742 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:51.415745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:51.415802 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:51.415881 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:51.416555 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.417374 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.418501 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.418861 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.418957 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.422212 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:51.422217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:51.422220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:51.422254 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:51.423736 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.424237 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.424241 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.424243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.424329 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.424590 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.424594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.424596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.424801 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:51.424892 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:51.425262 (+ 370us) server_negotiation.cc:299] Negotiation successful
0810 03:50:51.425398 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 03:50:51.425580 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.414981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:45527 (local address 127.0.0.1:39124)
0810 03:50:51.415345 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:51.415398 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:51.415583 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:51.415917 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:51.415922 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:51.416307 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:51.416446 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.416452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.417499 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.417503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.418385 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.418398 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.418998 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.419002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.419125 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.420033 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:51.420049 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:51.421996 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:51.423852 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.423864 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.423872 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.424131 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.424414 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.424418 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.424420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.424507 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.424908 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:51.424917 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:51.425048 (+ 131us) client_negotiation.cc:769] Sending connection context
0810 03:50:51.425188 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 03:50:51.425413 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":97,"threads_started":1}
I20260810 03:50:51.427341 6760 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.153.125:45527: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855"
member_type: NON_VOTER
last_known_addr {
host: "127.27.153.125"
port: 45527
}
attrs {
promote: true
}
}
dest_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b"
cas_config_opid_index: -1
I20260810 03:50:51.428126 6760 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: NON_VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.27.153.124@KRBTEST.COM; defaulting to no policy
I20260810 03:50:51.430163 7794 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5e655481b21e4dd4ac7c1135dbb4d855 (127.27.153.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: NON_VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: true } } }
I20260810 03:50:51.431233 7704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.153.124@KRBTEST.COM" created.
I20260810 03:50:51.433552 7794 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.153.125:45527
I20260810 03:50:51.433861 7794 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: NON_VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: true } } }
I20260810 03:50:51.433983 7794 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:51.434657 7704 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 1 FOLLOWER]: Refusing update from remote peer f3d2c2bbe823423ca1284d9fa2a52b2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 03:50:51.435523 7619 master.cc:537] Master@127.27.153.125:45527 shutting down...
I20260810 03:50:51.435531 7797 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: NON_VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 03:50:51.438179 6736 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260810 03:50:51.439883 6736 consensus_peers.cc:487] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b -> Peer 5e655481b21e4dd4ac7c1135dbb4d855 (127.27.153.125:45527): Couldn't send request to peer 5e655481b21e4dd4ac7c1135dbb4d855. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 03:50:51.441802 7619 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:51.442112 7619 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:51.442225 7619 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.153.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.124.keytab.
Entry for principal oryx/127.27.153.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.124.keytab.
I20260810 03:50:51.457304 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.124:39435
--webserver_interface=127.27.153.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.124.keytab
--principal=oryx/127.27.153.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:41933,127.27.153.125:45527,127.27.153.124:39435
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:50:51.457844 7619 master.cc:552] Master@127.27.153.125:45527 shutdown complete.
I20260810 03:50:51.457898 7619 master_runner.cc:305] Clearing existing system tablet
I20260810 03:50:51.458120 7619 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:51.458182 7619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:51.459610 7619 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 03:50:51.460579 7806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:51.460840 7619 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:51.460964 7619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 03:50:51.460995 7619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/wal
uuid: "5e655481b21e4dd4ac7c1135dbb4d855"
format_stamp: "Formatted at 2026-08-10 03:50:51 on dist-test-slave-pww0"
server_key: "f64ae5335180e1d30fd1661ebe886301"
server_key_iv: "619e951d6d4e27c0d5a8e7f7c0b86e05"
server_key_version: "encryptionkey@0"
I20260810 03:50:51.463406 7619 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:51.466272 7619 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 03:50:51.466398 7619 log.cc:1196] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 03:50:51.466684 7619 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855: Deleting consensus metadata
I20260810 03:50:51.466881 7619 master_runner.cc:315] Copying system tablet from 127.27.153.126:41933
I20260810 03:50:51.468518 7619 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855: tablet copy: Beginning tablet copy session from remote peer at address 127.27.153.126:41933
I20260810 03:50:51.480474 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.469284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.0.0.1:46422)
0810 03:50:51.471707 (+ 2423us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:51.471724 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:51.471877 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:51.472183 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:51.472186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:51.472410 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:51.472556 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.472563 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.473639 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.473643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.474622 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.474635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.475226 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.475230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.475336 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.476079 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:51.476094 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:51.476546 (+ 452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:51.478585 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.478589 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.478592 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.478868 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.479164 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.479167 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.479169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.479273 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.479853 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:51.479856 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:51.479962 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 03:50:51.480065 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 03:50:51.480248 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2327,"thread_start_us":101,"threads_started":1}
I20260810 03:50:51.480475 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.469399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46422 (local address 127.27.153.126:41933)
0810 03:50:51.469551 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:51.469557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:51.471839 (+ 2282us) server_negotiation.cc:407] Connection header received
0810 03:50:51.472001 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:51.472004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:51.472067 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:51.472150 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:51.472671 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.473524 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.474752 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.475107 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.475220 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.476716 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:51.476720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:51.476724 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:51.476756 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:51.478448 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.478966 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.478970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.478972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.479069 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.479393 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.479397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.479399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.479700 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:51.479779 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:51.480164 (+ 385us) server_negotiation.cc:299] Negotiation successful
0810 03:50:51.480311 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:50:51.481657 6797 tablet_copy_service.cc:140] P f3d2c2bbe823423ca1284d9fa2a52b2b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5e655481b21e4dd4ac7c1135dbb4d855 ({username='oryx', principal='oryx/127.27.153.125@KRBTEST.COM'} at 127.0.0.1:46422)
I20260810 03:50:51.481962 6797 tablet_copy_service.cc:161] P f3d2c2bbe823423ca1284d9fa2a52b2b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5e655481b21e4dd4ac7c1135dbb4d855 at {username='oryx', principal='oryx/127.27.153.125@KRBTEST.COM'} at 127.0.0.1:46422: session id = 5e655481b21e4dd4ac7c1135dbb4d855-00000000000000000000000000000000
I20260810 03:50:51.484126 6797 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 03:50:51.486194 7619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 03:50:51.488924 7619 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855: tablet copy: Starting download of 0 data blocks...
I20260810 03:50:51.489146 7619 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855: tablet copy: Starting download of 1 WAL segments...
I20260810 03:50:51.491503 7619 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 03:50:51.493883 6797 tablet_copy_service.cc:342] P f3d2c2bbe823423ca1284d9fa2a52b2b: Request end of tablet copy session 5e655481b21e4dd4ac7c1135dbb4d855-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.153.125@KRBTEST.COM'} at 127.0.0.1:46422
I20260810 03:50:51.494056 6797 tablet_copy_service.cc:433] P f3d2c2bbe823423ca1284d9fa2a52b2b: ending tablet copy session 5e655481b21e4dd4ac7c1135dbb4d855-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5e655481b21e4dd4ac7c1135dbb4d855
I20260810 03:50:51.502911 7619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:51.505044 7820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:51.508661 7821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:51.508737 7824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:51.508929 7619 server_base.cc:1034] running on GCE node
I20260810 03:50:51.509207 7619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:51.509541 7619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:51.510692 7619 hybrid_clock.cc:648] HybridClock initialized: now 1786333851510669 us; error 41 us; skew 500 ppm
Aug 10 03:50:51 dist-test-slave-pww0 krb5kdc[6667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333851, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:51.512503 7619 init.cc:377] Logged in from keytab as oryx/127.27.153.125@KRBTEST.COM (short username oryx)
I20260810 03:50:51.513238 7619 webserver.cc:460] Webserver started at http://127.27.153.125:45413/ using document root <none> and password file <none>
I20260810 03:50:51.513443 7619 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:51.513487 7619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:51.514904 7619 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 03:50:51.516052 7840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:51.516346 7619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 03:50:51.516428 7619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/wal
uuid: "5e655481b21e4dd4ac7c1135dbb4d855"
format_stamp: "Formatted at 2026-08-10 03:50:51 on dist-test-slave-pww0"
server_key: "f64ae5335180e1d30fd1661ebe886301"
server_key_iv: "619e951d6d4e27c0d5a8e7f7c0b86e05"
server_key_version: "encryptionkey@0"
I20260810 03:50:51.516515 7619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:51.545248 7619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:51.547557 7619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:51.552737 7619 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.125:45527
I20260810 03:50:51.553542 7619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-1/data/info.pb
I20260810 03:50:51.554095 7933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.125:45527 every 8 connection(s)
I20260810 03:50:51.555688 7934 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:50:51.556775 7934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855: Bootstrap starting.
I20260810 03:50:51.567780 7934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:50:51.568087 7934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855: Bootstrap complete.
I20260810 03:50:51.568665 7934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: NON_VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: true } }
I20260810 03:50:51.568792 7934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5e655481b21e4dd4ac7c1135dbb4d855, State: Initialized, Role: LEARNER
I20260810 03:50:51.568881 7934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: NON_VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: true } }
I20260810 03:50:51.569239 7942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: NON_VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: true } } }
I20260810 03:50:51.569375 7942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [sys.catalog]: This master's current role is: LEARNER
I20260810 03:50:51.569377 7934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:50:51.572273 7962 catalog_manager.cc:1259] Loaded cluster ID: f35189ad36ec45d1a449e600cbd8824f
I20260810 03:50:51.572331 7962 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855: loading cluster ID for follower catalog manager: success
I20260810 03:50:51.573894 7962 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855: acquiring CA information for follower catalog manager: success
I20260810 03:50:51.575039 7962 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 03:50:51.599057 7804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:51.599330 7804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:51.599391 7804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:51.603552 7804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:50:51.603627 7804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:51.603659 7804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:51.603693 7804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:50:51.603721 7804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:50:51.609373 7804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:41933,127.27.153.125:45527,127.27.153.124:39435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.124.keytab
--principal=oryx/127.27.153.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.124:39435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.153.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:51.610572 7804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:51.611716 7804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:51.620911 7969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:51.621032 7117 heartbeater.cc:503] Master 127.27.153.126:41933 was elected leader, sending a full tablet report...
W20260810 03:50:51.620949 7972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:51.621109 7804 server_base.cc:1034] running on GCE node
W20260810 03:50:51.621362 7974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:51.622150 7804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:51.623426 7804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:51.624627 7804 hybrid_clock.cc:648] HybridClock initialized: now 1786333851624607 us; error 31 us; skew 500 ppm
Aug 10 03:50:51 dist-test-slave-pww0 krb5kdc[6667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333851, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:51.627357 7804 init.cc:377] Logged in from keytab as oryx/127.27.153.124@KRBTEST.COM (short username oryx)
I20260810 03:50:51.628366 7804 webserver.cc:460] Webserver started at http://127.27.153.124:43327/ using document root <none> and password file <none>
I20260810 03:50:51.628893 7804 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:51.628952 7804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:51.629128 7804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:51.631304 7804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/data/instance:
uuid: "0fed699bb95348be8b9a62282240a7de"
format_stamp: "Formatted at 2026-08-10 03:50:51 on dist-test-slave-pww0"
server_key: "95299993e68583ca04c42e9745d16136"
server_key_iv: "35d9ea5467b724556c5b2fd12ea0873e"
server_key_version: "encryptionkey@0"
I20260810 03:50:51.631910 7804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/wal/instance:
uuid: "0fed699bb95348be8b9a62282240a7de"
format_stamp: "Formatted at 2026-08-10 03:50:51 on dist-test-slave-pww0"
server_key: "95299993e68583ca04c42e9745d16136"
server_key_iv: "35d9ea5467b724556c5b2fd12ea0873e"
server_key_version: "encryptionkey@0"
I20260810 03:50:51.635931 7804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:50:51.638612 7987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:51.640069 7804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:51.640226 7804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/wal
uuid: "0fed699bb95348be8b9a62282240a7de"
format_stamp: "Formatted at 2026-08-10 03:50:51 on dist-test-slave-pww0"
server_key: "95299993e68583ca04c42e9745d16136"
server_key_iv: "35d9ea5467b724556c5b2fd12ea0873e"
server_key_version: "encryptionkey@0"
I20260810 03:50:51.640350 7804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:51.659135 7804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:51.662876 7804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:51.663084 7804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:51.670746 7804 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.124:39435
I20260810 03:50:51.671008 8079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.124:39435 every 8 connection(s)
I20260810 03:50:51.671901 7804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/data/info.pb
I20260810 03:50:51.674890 8081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:51.676494 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 7804
I20260810 03:50:51.679044 8081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:50:51 dist-test-slave-pww0 krb5kdc[6667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.153.124@KRBTEST.COM
Aug 10 03:50:51 dist-test-slave-pww0 krb5kdc[6667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333851, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.124@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM
I20260810 03:50:51.691018 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.681307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.27.153.126:41933)
0810 03:50:51.681445 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:51.681449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:51.681587 (+ 138us) server_negotiation.cc:407] Connection header received
0810 03:50:51.681759 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:51.681762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:51.681812 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:51.681874 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:51.682684 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.683530 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.684799 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.685075 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.685176 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.687869 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:51.687874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:51.687877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:51.687915 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:51.689184 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.689653 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.689656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.689658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.689738 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.690033 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.690037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.690039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.690240 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:51.690339 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:51.690716 (+ 377us) server_negotiation.cc:299] Negotiation successful
0810 03:50:51.690844 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 03:50:51.691948 8086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.681103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.0.0.1:46432)
0810 03:50:51.681455 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:51.681506 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:51.681643 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:51.681937 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:51.681943 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:51.682385 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:51.682535 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.682543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.683690 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.683693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.684671 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.684680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.685241 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.685245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.685326 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.685770 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:51.685787 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:51.687716 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:51.689302 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.689312 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.689323 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.689553 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.689822 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.689825 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.689827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.689940 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.690348 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:51.690356 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:51.690505 (+ 149us) client_negotiation.cc:769] Sending connection context
0810 03:50:51.690659 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 03:50:51.691006 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":34,"thread_start_us":103,"threads_started":1}
I20260810 03:50:51.691952 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.678160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48680 (local address 127.27.153.124:39435)
0810 03:50:51.678620 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:51.678637 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:51.678674 (+ 37us) server_negotiation.cc:407] Connection header received
0810 03:50:51.679360 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:51.679379 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:51.679781 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:51.680110 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:51.680940 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.682400 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.683682 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.684459 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.684624 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.687341 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:51.687347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:51.687358 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:51.687383 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:51.689121 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.689578 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.689584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.689585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.689704 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.689976 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.689980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.689981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.690394 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:51.690603 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:51.690823 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 03:50:51.691196 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":209,"server-negotiator.queue_time_us":311,"thread_start_us":151,"threads_started":1}
I20260810 03:50:51.693956 8081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:50:51 dist-test-slave-pww0 krb5kdc[6667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333851, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.124@KRBTEST.COM for oryx/127.27.153.125@KRBTEST.COM
I20260810 03:50:51.702617 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.694641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39126 (local address 127.27.153.125:45527)
0810 03:50:51.694949 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:51.694952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:51.694964 (+ 12us) server_negotiation.cc:407] Connection header received
0810 03:50:51.695003 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:51.695005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:51.695045 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:51.695121 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:51.695555 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.696365 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.697195 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.697451 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.697532 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.699646 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:51.699649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:51.699651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:51.699679 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:51.700932 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.701391 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.701394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.701395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.701485 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.701840 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.701843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.701844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.702056 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:51.702142 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:51.702353 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 03:50:51.702471 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":142,"threads_started":1}
I20260810 03:50:51.702617 8086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.694512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:45527 (local address 127.0.0.1:39126)
0810 03:50:51.694639 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:51.694654 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:51.694734 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:51.695150 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:51.695152 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:51.695315 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:51.695415 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.695421 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.696481 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.696484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.697096 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.697105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.697556 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.697559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.697643 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.698093 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:51.698105 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:51.699399 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:51.701058 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.701060 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.701062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.701269 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.701626 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.701628 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.701630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.701727 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.702135 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:51.702137 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:51.702224 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 03:50:51.702301 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 03:50:51.702471 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 03:50:51.703706 8081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:50:51 dist-test-slave-pww0 krb5kdc[6667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333851, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.124@KRBTEST.COM for oryx/127.27.153.124@KRBTEST.COM
I20260810 03:50:51.712097 8086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.704218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:39435 (local address 127.0.0.1:48688)
0810 03:50:51.704362 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:51.704376 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:51.704456 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:51.704714 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:51.704717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:51.704864 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:51.704944 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.704948 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.706192 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.706195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.706720 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.706728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.707186 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.707189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.707245 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.707734 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:51.707745 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:51.708969 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:51.710640 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.710643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.710644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.710836 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.711063 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.711066 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.711067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.711146 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.711603 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:51.711605 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:51.711695 (+ 90us) client_negotiation.cc:769] Sending connection context
0810 03:50:51.711784 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 03:50:51.711928 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 03:50:51.712319 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.704292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48688 (local address 127.27.153.124:39435)
0810 03:50:51.704408 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:51.704412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:51.704424 (+ 12us) server_negotiation.cc:407] Connection header received
0810 03:50:51.704566 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:51.704569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:51.704610 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:51.704684 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:51.705063 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.705699 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.706831 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.707073 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.707172 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.709118 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:51.709121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:51.709123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:51.709161 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:51.710519 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.710926 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.710929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.710930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.711000 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.711263 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.711274 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.711275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.711517 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:51.711609 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:51.711834 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 03:50:51.712039 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":53}
I20260810 03:50:51.715484 8081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de: Bootstrap starting.
I20260810 03:50:51.718112 8081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:51.718914 8081 log.cc:824] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:51.721338 8081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de: No bootstrap required, opened a new log
I20260810 03:50:51.724648 8081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } }
I20260810 03:50:51.724936 8081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:51.725006 8081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fed699bb95348be8b9a62282240a7de, State: Initialized, Role: FOLLOWER
I20260810 03:50:51.725461 8081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } }
I20260810 03:50:51.725996 8092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } } }
I20260810 03:50:51.726142 8092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:50:51.726753 8081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [sys.catalog]: configured and running, proceeding with master startup.
W20260810 03:50:51.731724 8106 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:50:51.731813 8106 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:50:51.735579 7804 master_runner.cc:428] Detected that this master 0fed699bb95348be8b9a62282240a7de is joining an existing cluster
I20260810 03:50:51.735662 7804 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.153.124:39435
I20260810 03:50:51.745571 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 6883
I20260810 03:50:51.763228 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:35535
--local_ip_for_outbound_sockets=127.27.153.65
--tserver_master_addrs=127.27.153.126:41933
--webserver_port=37765
--webserver_interface=127.27.153.65
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.65.keytab
--principal=oryx/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.153.126:41933,127.27.153.125:45527
--tserver_master_addrs=127.27.153.126:41933,127.27.153.125:45527,127.27.153.124:39435 with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:50:51.782280 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.738380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48324 (local address 127.27.153.126:41933)
0810 03:50:51.738574 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:51.738580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:51.738942 (+ 362us) server_negotiation.cc:407] Connection header received
0810 03:50:51.739319 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:51.739323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:51.739382 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:51.739515 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:51.740028 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.740890 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.741756 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.742088 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.742200 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.743831 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:51.743855 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:51.743858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:51.743886 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:51.745830 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.780271 (+ 34441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.780277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.780279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.780416 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.781149 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.781153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.781155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.781445 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:51.781528 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:51.781935 (+ 407us) server_negotiation.cc:299] Negotiation successful
0810 03:50:51.782086 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:50:51.783047 8119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.738296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.0.0.1:48324)
0810 03:50:51.738843 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:51.738858 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:51.739176 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:51.739557 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:51.739560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:51.739780 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:51.739904 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.739911 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.741001 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.741004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.741634 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.741643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.742211 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.742214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.742286 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.743227 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:51.743240 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:51.743683 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:51.747706 (+ 4023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.747710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.747713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.780050 (+ 32337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.780558 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.780561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.780564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.781005 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.781611 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:51.781614 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:51.781721 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 03:50:51.782679 (+ 958us) client_negotiation.cc:240] Negotiation successful
0810 03:50:51.782857 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":111,"threads_started":1}
I20260810 03:50:51.786567 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.738737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40108 (local address 127.27.153.124:39435)
0810 03:50:51.738820 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:51.738825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:51.743777 (+ 4952us) server_negotiation.cc:407] Connection header received
0810 03:50:51.743915 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:51.743918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:51.743964 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:51.744034 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:51.744739 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.745386 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.746666 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.747067 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.747149 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.781347 (+ 34198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:51.781353 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:51.781356 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:51.781383 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:51.783421 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.784293 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.784297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.784299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.784382 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.784673 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.784676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.784678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.785023 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:51.785121 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:51.785430 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 03:50:51.785649 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 03:50:51.788254 8123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.738320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:39435 (local address 127.0.0.1:40108)
0810 03:50:51.743689 (+ 5369us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:51.743705 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:51.743821 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:51.744152 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:51.744155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:51.744426 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:51.744620 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.744630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.745528 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.745531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.746512 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.746525 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.747328 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.747331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.747439 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.748943 (+ 1504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:51.748954 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:51.781146 (+ 32192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:51.783876 (+ 2730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.783880 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.783882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.784171 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.784486 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.784490 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.784492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.784576 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.785187 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:51.785190 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:51.785276 (+ 86us) client_negotiation.cc:769] Sending connection context
0810 03:50:51.787083 (+ 1807us) client_negotiation.cc:240] Negotiation successful
0810 03:50:51.787275 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5279,"thread_start_us":130,"threads_started":1}
I20260810 03:50:51.791175 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.739008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47676 (local address 127.27.153.125:45527)
0810 03:50:51.739139 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:51.739143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:51.744421 (+ 5278us) server_negotiation.cc:407] Connection header received
0810 03:50:51.744942 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:51.744945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:51.745011 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:51.745069 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:51.757853 (+ 12784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.758737 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.760077 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.760494 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.760642 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.786138 (+ 25496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:51.786142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:51.786145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:51.786179 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:51.787964 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.789773 (+ 1809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.789777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.789779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.789878 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.790188 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.790192 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.790194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.790457 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:51.790544 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:51.790878 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 03:50:51.791026 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 03:50:51.791728 8126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.738894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:45527 (local address 127.0.0.1:47676)
0810 03:50:51.744077 (+ 5183us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:51.744089 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:51.744509 (+ 420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:51.757201 (+ 12692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:51.757205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:51.757478 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:51.757653 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.757663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.758911 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.758914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.759892 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.759905 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.760670 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.760674 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.760772 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.761379 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:51.761396 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:51.785945 (+ 24549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:51.789388 (+ 3443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.789391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.789393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.789640 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.789988 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.789991 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.789993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.790084 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.790604 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:51.790607 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:51.790701 (+ 94us) client_negotiation.cc:769] Sending connection context
0810 03:50:51.791387 (+ 686us) client_negotiation.cc:240] Negotiation successful
0810 03:50:51.791563 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5112,"thread_start_us":105,"threads_started":1}
Aug 10 03:50:51 dist-test-slave-pww0 krb5kdc[6667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333850, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.126@KRBTEST.COM for oryx/127.27.153.124@KRBTEST.COM
I20260810 03:50:51.798910 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.788881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.27.153.124:39435)
0810 03:50:51.789219 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:51.789223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:51.789236 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:50:51.789302 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:51.789305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:51.789354 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:51.789423 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:51.790071 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.790883 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.792526 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.792862 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.792961 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.795374 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:51.795378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:51.795381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:51.795407 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:51.797159 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.797653 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.797657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.797659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.797747 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.798027 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.798030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.798032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.798265 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:51.798358 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:51.798628 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 03:50:51.798753 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260810 03:50:51.798915 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.788400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:39435 (local address 127.0.0.1:40124)
0810 03:50:51.788658 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:51.788672 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:51.788750 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:51.789553 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:51.789557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:51.789758 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:51.789875 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.789882 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.791018 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.791022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.791935 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.791947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.792991 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.792995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.793103 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.793645 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:51.793660 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:51.795211 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:51.797285 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.797289 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.797291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.797552 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.797844 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.797847 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.797849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.797943 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.798353 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:51.798356 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:51.798468 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 03:50:51.798570 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 03:50:51.798738 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180}
I20260810 03:50:51.800469 6760 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.153.124:39435: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0fed699bb95348be8b9a62282240a7de"
member_type: NON_VOTER
last_known_addr {
host: "127.27.153.124"
port: 39435
}
attrs {
promote: true
}
}
dest_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b"
cas_config_opid_index: 5
I20260810 03:50:51.801105 6760 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: NON_VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: true } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: NON_VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: true } }
I20260810 03:50:51.802587 7797 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0fed699bb95348be8b9a62282240a7de (127.27.153.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: NON_VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: true } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: NON_VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: true } } }
I20260810 03:50:51.804227 7797 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.153.124:39435
I20260810 03:50:51.804427 7797 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: NON_VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: true } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: NON_VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: true } } }
I20260810 03:50:51.804534 7797 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:51.805392 8040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:51.806842 8040 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 1 FOLLOWER]: Refusing update from remote peer f3d2c2bbe823423ca1284d9fa2a52b2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 03:50:51.807258 7804 master.cc:537] Master@127.27.153.124:39435 shutting down...
I20260810 03:50:51.808104 7794 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: NON_VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 03:50:51.810130 6731 consensus_peers.cc:487] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b -> Peer 0fed699bb95348be8b9a62282240a7de (127.27.153.124:39435): Couldn't send request to peer 0fed699bb95348be8b9a62282240a7de. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 03:50:51.814945 7804 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:51.815281 7804 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:51.815382 7804 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de: stopping tablet replica
I20260810 03:50:51.819422 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.803117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47682 (local address 127.27.153.125:45527)
0810 03:50:51.803341 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:51.803346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:51.814147 (+ 10801us) server_negotiation.cc:407] Connection header received
0810 03:50:51.814310 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:51.814313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:51.814388 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:51.814485 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:51.814902 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.815746 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.817751 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.818437 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.818541 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.818692 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:51.819115 (+ 423us) server_negotiation.cc:299] Negotiation successful
0810 03:50:51.819266 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260810 03:50:51.820115 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.802822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:45527 (local address 127.0.0.1:47682)
0810 03:50:51.814044 (+ 11222us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:51.814065 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:51.814197 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:51.814637 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:51.814640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:51.814662 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:51.814767 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.814774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.816089 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.816093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.817558 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.817571 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.818808 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.818811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.818915 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.818974 (+ 59us) client_negotiation.cc:769] Sending connection context
0810 03:50:51.819840 (+ 866us) client_negotiation.cc:240] Negotiation successful
0810 03:50:51.819953 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8557}
I20260810 03:50:51.821010 7875 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 1 LEARNER]: Refusing update from remote peer f3d2c2bbe823423ca1284d9fa2a52b2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 03:50:51.821848 7804 master.cc:552] Master@127.27.153.124:39435 shutdown complete.
I20260810 03:50:51.821903 7804 master_runner.cc:305] Clearing existing system tablet
I20260810 03:50:51.822147 7804 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:51.822207 7804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:51.823323 8130 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: NON_VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 03:50:51.823855 7804 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 03:50:51.825551 7875 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0fed699bb95348be8b9a62282240a7de (127.27.153.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: NON_VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: true } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: NON_VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: true } } }
I20260810 03:50:51.827615 7942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3d2c2bbe823423ca1284d9fa2a52b2b. Latest consensus state: current_term: 1 leader_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: NON_VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: true } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: NON_VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: true } } }
I20260810 03:50:51.827756 7942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [sys.catalog]: This master's current role is: LEARNER
I20260810 03:50:51.827834 7942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: NON_VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: true } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: NON_VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: true } } }
I20260810 03:50:51.827894 7942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [sys.catalog]: This master's current role is: LEARNER
I20260810 03:50:51.828647 8134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:51.828836 7794 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b: attempting to promote NON_VOTER 5e655481b21e4dd4ac7c1135dbb4d855 to VOTER
I20260810 03:50:51.828961 7804 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:51.829253 7804 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260810 03:50:51.829341 7804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/wal
uuid: "0fed699bb95348be8b9a62282240a7de"
format_stamp: "Formatted at 2026-08-10 03:50:51 on dist-test-slave-pww0"
server_key: "95299993e68583ca04c42e9745d16136"
server_key_iv: "35d9ea5467b724556c5b2fd12ea0873e"
server_key_version: "encryptionkey@0"
I20260810 03:50:51.829344 7794 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: NON_VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: true } }
I20260810 03:50:51.831158 7875 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 1 LEARNER]: Refusing update from remote peer f3d2c2bbe823423ca1284d9fa2a52b2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 03:50:51.831841 7797 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 03:50:51.833155 7804 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:51.834030 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.833692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:39435 (local address 127.0.0.1:40140)
0810 03:50:51.833860 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:51.833948 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260810 03:50:51.834888 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.834614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:39435 (local address 127.0.0.1:40154)
0810 03:50:51.834763 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:51.834818 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:39435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260810 03:50:51.835021 6731 consensus_peers.cc:487] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b -> Peer 0fed699bb95348be8b9a62282240a7de (127.27.153.124:39435): Couldn't send request to peer 0fed699bb95348be8b9a62282240a7de. Status: Network error: Client connection negotiation failed: client connection to 127.27.153.124:39435: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 03:50:51.836090 7794 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5e655481b21e4dd4ac7c1135dbb4d855 (127.27.153.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: NON_VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: true } } }
I20260810 03:50:51.837397 7804 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 03:50:51.837558 7804 log.cc:1196] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 03:50:51.837936 7804 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de: Deleting consensus metadata
I20260810 03:50:51.838275 7804 master_runner.cc:315] Copying system tablet from 127.27.153.126:41933
I20260810 03:50:51.838508 7794 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: NON_VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: true } } }
I20260810 03:50:51.838649 7794 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:51.839174 7797 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: NON_VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: true } } }
I20260810 03:50:51.839300 7797 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:51.839659 7875 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 5e655481b21e4dd4ac7c1135dbb4d855 (127.27.153.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: NON_VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: true } } }
I20260810 03:50:51.840049 7804 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de: tablet copy: Beginning tablet copy session from remote peer at address 127.27.153.126:41933
I20260810 03:50:51.841321 8141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: NON_VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: true } } }
I20260810 03:50:51.841473 8141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:50:51.856611 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.844002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48338 (local address 127.27.153.126:41933)
0810 03:50:51.844141 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:51.844146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:51.847881 (+ 3735us) server_negotiation.cc:407] Connection header received
0810 03:50:51.848115 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:51.848120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:51.848186 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:51.848403 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:51.849036 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.849846 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.851115 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.851483 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.851597 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.852934 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:51.852938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:51.852941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:51.852972 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:51.854659 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.855222 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.855226 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.855228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.855332 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:51.855663 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:51.855666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:51.855668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:51.855919 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:51.856029 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:51.856324 (+ 295us) server_negotiation.cc:299] Negotiation successful
0810 03:50:51.856456 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:50:51.856604 8152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:51.843902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.0.0.1:48338)
0810 03:50:51.847738 (+ 3836us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:51.847759 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:51.847915 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:51.848410 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:51.848413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:51.848672 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:51.848896 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.848908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.850017 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.850021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.850971 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:51.850983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:51.851612 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:51.851616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:51.851715 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:51.852299 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:51.852315 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:51.852772 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:51.854804 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.854807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.854809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.855102 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.855432 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:51.855436 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:51.855438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:51.855567 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:51.856024 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:51.856027 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:51.856144 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 03:50:51.856254 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 03:50:51.856413 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3723,"thread_start_us":133,"threads_started":1}
I20260810 03:50:51.857365 6797 tablet_copy_service.cc:140] P f3d2c2bbe823423ca1284d9fa2a52b2b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0fed699bb95348be8b9a62282240a7de ({username='oryx', principal='oryx/127.27.153.124@KRBTEST.COM'} at 127.0.0.1:48338)
I20260810 03:50:51.857478 6797 tablet_copy_service.cc:161] P f3d2c2bbe823423ca1284d9fa2a52b2b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0fed699bb95348be8b9a62282240a7de at {username='oryx', principal='oryx/127.27.153.124@KRBTEST.COM'} at 127.0.0.1:48338: session id = 0fed699bb95348be8b9a62282240a7de-00000000000000000000000000000000
I20260810 03:50:51.858632 6797 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 03:50:51.859148 7364 heartbeater.cc:503] Master 127.27.153.126:41933 was elected leader, sending a full tablet report...
I20260810 03:50:51.862089 7804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 03:50:51.865042 7804 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de: tablet copy: Starting download of 0 data blocks...
I20260810 03:50:51.865396 7804 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de: tablet copy: Starting download of 1 WAL segments...
I20260810 03:50:51.867210 7804 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 03:50:51.869522 6797 tablet_copy_service.cc:342] P f3d2c2bbe823423ca1284d9fa2a52b2b: Request end of tablet copy session 0fed699bb95348be8b9a62282240a7de-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.153.124@KRBTEST.COM'} at 127.0.0.1:48338
I20260810 03:50:51.869627 6797 tablet_copy_service.cc:433] P f3d2c2bbe823423ca1284d9fa2a52b2b: ending tablet copy session 0fed699bb95348be8b9a62282240a7de-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0fed699bb95348be8b9a62282240a7de
I20260810 03:50:51.878623 7804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:51.880523 8157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:51.881105 8160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:51.881362 8162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:51.881429 7804 server_base.cc:1034] running on GCE node
I20260810 03:50:51.881675 7804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:51.882077 7804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:51.883361 7804 hybrid_clock.cc:648] HybridClock initialized: now 1786333851883345 us; error 34 us; skew 500 ppm
Aug 10 03:50:51 dist-test-slave-pww0 krb5kdc[6667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333851, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:51.885280 7804 init.cc:377] Logged in from keytab as oryx/127.27.153.124@KRBTEST.COM (short username oryx)
I20260810 03:50:51.885829 7804 webserver.cc:460] Webserver started at http://127.27.153.124:35919/ using document root <none> and password file <none>
I20260810 03:50:51.886029 7804 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:51.886078 7804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:51.887656 7804 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 03:50:51.888823 8171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:51.889158 7804 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 03:50:51.889245 7804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/wal
uuid: "0fed699bb95348be8b9a62282240a7de"
format_stamp: "Formatted at 2026-08-10 03:50:51 on dist-test-slave-pww0"
server_key: "95299993e68583ca04c42e9745d16136"
server_key_iv: "35d9ea5467b724556c5b2fd12ea0873e"
server_key_version: "encryptionkey@0"
I20260810 03:50:51.889341 7804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:51.909821 7804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:51.912070 7804 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260810 03:50:51.912976 8129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:51.913257 8129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:51.913332 8129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 03:50:51.917892 7804 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.124:39435
I20260810 03:50:51.918068 8262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.124:39435 every 8 connection(s)
I20260810 03:50:51.918531 7804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/master-2/data/info.pb
W20260810 03:50:51.919008 8129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:51.919075 8129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:51.919166 8129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:50:51.920819 8267 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:50:51.921876 8267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de: Bootstrap starting.
I20260810 03:50:51.925328 8129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.65.keytab
--principal=oryx/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:35535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=37765
--tserver_master_addrs=127.27.153.126:41933,127.27.153.125:45527,127.27.153.124:39435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:51.926803 8129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:51.927968 8129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:50:51.933089 8267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:50:51.933408 8267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de: Bootstrap complete.
I20260810 03:50:51.933996 8267 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: NON_VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: true } }
I20260810 03:50:51.934163 8267 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0fed699bb95348be8b9a62282240a7de, State: Initialized, Role: LEARNER
I20260810 03:50:51.934285 8267 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: NON_VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: true } }
I20260810 03:50:51.934609 8277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: NON_VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: true } } }
I20260810 03:50:51.934767 8267 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:50:51.934772 8277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [sys.catalog]: This master's current role is: LEARNER
W20260810 03:50:51.938460 8274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:51.938520 8276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:51.938061 8297 catalog_manager.cc:1259] Loaded cluster ID: f35189ad36ec45d1a449e600cbd8824f
W20260810 03:50:51.938552 8273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:51.938722 8129 server_base.cc:1034] running on GCE node
I20260810 03:50:51.938865 8297 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de: loading cluster ID for follower catalog manager: success
I20260810 03:50:51.939296 8129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:51.940070 8129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:51.940713 8297 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de: acquiring CA information for follower catalog manager: success
I20260810 03:50:51.941251 8129 hybrid_clock.cc:648] HybridClock initialized: now 1786333851941233 us; error 32 us; skew 500 ppm
I20260810 03:50:51.941908 8297 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Aug 10 03:50:51 dist-test-slave-pww0 krb5kdc[6667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333851, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:51.944252 8129 init.cc:377] Logged in from keytab as oryx/127.27.153.65@KRBTEST.COM (short username oryx)
I20260810 03:50:51.945369 8129 webserver.cc:460] Webserver started at http://127.27.153.65:37765/ using document root <none> and password file <none>
I20260810 03:50:51.945983 8129 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:51.946066 8129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:51.950282 8129 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260810 03:50:51.952648 8306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:51.954049 8129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:51.954172 8129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "6720a63653bd424bba05965f0b0844dc"
format_stamp: "Formatted at 2026-08-10 03:50:50 on dist-test-slave-pww0"
server_key: "1e241a55e328c88fd30358594b68e330"
server_key_iv: "bd625c001f92dd96185b1d007db4ef3a"
server_key_version: "encryptionkey@0"
I20260810 03:50:51.954806 8129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:51.988952 8129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:51.992758 8129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:51.992929 8129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:51.993536 8129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:51.994426 8129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:51.994477 8129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:51.994517 8129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:51.994539 8129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:52.004598 8129 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:35535
I20260810 03:50:52.004628 8506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:35535 every 8 connection(s)
I20260810 03:50:52.005710 8129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:50:52.006008 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 8129
I20260810 03:50:52.006219 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 7127
Aug 10 03:50:52 dist-test-slave-pww0 krb5kdc[6667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333851, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM
I20260810 03:50:52.035795 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.010104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:33117 (local address 127.27.153.126:41933)
0810 03:50:52.010258 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:52.010264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:52.017231 (+ 6967us) server_negotiation.cc:407] Connection header received
0810 03:50:52.019520 (+ 2289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:52.019525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:52.019601 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:52.019666 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:52.022283 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.023340 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.025647 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.026069 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.026173 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.030369 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:52.030374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:52.030377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:52.030418 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:52.032488 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.033388 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.033393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.033395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.033509 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.034020 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.034024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.034026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.034343 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:52.034442 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:52.035387 (+ 945us) server_negotiation.cc:299] Negotiation successful
0810 03:50:52.035598 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 03:50:52.038060 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.008281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.65:33117)
0810 03:50:52.016942 (+ 8661us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:52.017063 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:52.018807 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:52.020343 (+ 1536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:52.020357 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:52.021067 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:52.022032 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.022059 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.023573 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.023577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.025454 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.025469 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.026448 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.026454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.026732 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.027624 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:52.027650 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:52.030036 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:52.032687 (+ 2651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:52.032696 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:52.032720 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:52.033237 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:52.033644 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:52.033648 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:52.033650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:52.033891 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:52.034526 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:52.034534 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:52.034998 (+ 464us) client_negotiation.cc:769] Sending connection context
0810 03:50:52.036096 (+ 1098us) client_negotiation.cc:240] Negotiation successful
0810 03:50:52.036619 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8208,"thread_start_us":686,"threads_started":1}
I20260810 03:50:52.043610 8508 heartbeater.cc:347] Connected to a master server at 127.27.153.126:41933
I20260810 03:50:52.043922 8508 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:52.044607 8508 heartbeater.cc:511] Master 127.27.153.126:41933 requested a full tablet report, sending...
I20260810 03:50:52.046216 6760 ts_manager.cc:194] Re-registered known tserver with Master: 6720a63653bd424bba05965f0b0844dc (127.27.153.65:35535)
I20260810 03:50:52.047017 6760 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:33117
Aug 10 03:50:52 dist-test-slave-pww0 krb5kdc[6667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333851, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.125@KRBTEST.COM
I20260810 03:50:52.049208 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:41001
--local_ip_for_outbound_sockets=127.27.153.66
--tserver_master_addrs=127.27.153.126:41933
--webserver_port=46207
--webserver_interface=127.27.153.66
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.66.keytab
--principal=oryx/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.153.126:41933,127.27.153.125:45527
--tserver_master_addrs=127.27.153.126:41933,127.27.153.125:45527,127.27.153.124:39435 with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:50:52 dist-test-slave-pww0 krb5kdc[6667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333851, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.124@KRBTEST.COM
I20260810 03:50:52.052747 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.009750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:45527 (local address 127.27.153.65:52753)
0810 03:50:52.042045 (+ 32295us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:52.042064 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:52.042282 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:52.042703 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:52.042706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:52.042919 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:52.043062 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.043069 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.044293 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.044296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.045108 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.045120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.045894 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.045897 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.046006 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.046562 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:52.046577 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:52.048707 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:52.050726 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:52.050730 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:52.050732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:52.051063 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:52.051438 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:52.051443 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:52.051445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:52.051607 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:52.052084 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:52.052088 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:52.052187 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 03:50:52.052331 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 03:50:52.052540 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32210,"thread_start_us":2054,"threads_started":1}
I20260810 03:50:52.052840 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.012213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:52753 (local address 127.27.153.125:45527)
0810 03:50:52.012361 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:52.012366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:52.042172 (+ 29806us) server_negotiation.cc:407] Connection header received
0810 03:50:52.042414 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:52.042418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:52.042482 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:52.042539 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:52.043227 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.044104 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.045271 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.045625 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.045712 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.048894 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:52.048898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:52.048901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:52.048932 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:52.050597 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.051176 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.051181 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.051182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.051295 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.051730 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.051734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.051735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.051959 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:52.052081 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:52.052438 (+ 357us) server_negotiation.cc:299] Negotiation successful
0810 03:50:52.052593 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 03:50:52.053879 8509 heartbeater.cc:347] Connected to a master server at 127.27.153.125:45527
I20260810 03:50:52.054001 8509 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:52.054284 8509 heartbeater.cc:511] Master 127.27.153.125:45527 requested a full tablet report, sending...
I20260810 03:50:52.054489 8515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.016777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:39435 (local address 127.27.153.65:34155)
0810 03:50:52.042522 (+ 25745us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:52.042536 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:52.042633 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:52.043438 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:52.043440 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:52.043686 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:52.043908 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.043916 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.045465 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.045469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.046595 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.046606 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.047993 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.047996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.048124 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.048617 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:52.048641 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:52.050146 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:52.052281 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:52.052286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:52.052288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:52.052593 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:52.052979 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:52.052982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:52.052984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:52.053082 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:52.053871 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:52.053874 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:52.053972 (+ 98us) client_negotiation.cc:769] Sending connection context
0810 03:50:52.054106 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 03:50:52.054293 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25655,"thread_start_us":2007,"threads_started":1}
I20260810 03:50:52.054483 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.019397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:34155 (local address 127.27.153.124:39435)
0810 03:50:52.043106 (+ 23709us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:52.043111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:52.043125 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:50:52.043177 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:52.043180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:52.043228 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:52.043311 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:52.044144 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.045277 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.046757 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.047104 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.047985 (+ 881us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.050336 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:52.050340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:52.050342 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:52.050376 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:52.052169 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.052728 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.052732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.052734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.052845 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.053275 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.053279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.053281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.053666 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:52.053779 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:52.054191 (+ 412us) server_negotiation.cc:299] Negotiation successful
0810 03:50:52.054320 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23618,"thread_start_us":1756,"threads_started":1}
I20260810 03:50:52.055668 8507 heartbeater.cc:347] Connected to a master server at 127.27.153.124:39435
I20260810 03:50:52.055676 7857 ts_manager.cc:194] Registered new tserver with Master: 6720a63653bd424bba05965f0b0844dc (127.27.153.65:35535)
I20260810 03:50:52.055788 8507 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:52.055970 8507 heartbeater.cc:511] Master 127.27.153.124:39435 requested a full tablet report, sending...
I20260810 03:50:52.057065 8191 ts_manager.cc:194] Registered new tserver with Master: 6720a63653bd424bba05965f0b0844dc (127.27.153.65:35535)
I20260810 03:50:52.107712 7607 heartbeater.cc:503] Master 127.27.153.126:41933 was elected leader, sending a full tablet report...
W20260810 03:50:52.171788 8518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:52.172091 8518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:52.172137 8518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:52.176242 8518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:52.176316 8518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:52.176378 8518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:50:52.181834 8518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.66.keytab
--principal=oryx/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:41001
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=46207
--tserver_master_addrs=127.27.153.126:41933,127.27.153.125:45527,127.27.153.124:39435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:52.182948 8518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:52.183979 8518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:52.194068 8529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:52.194078 8531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:52.194087 8534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:52.194478 8518 server_base.cc:1034] running on GCE node
I20260810 03:50:52.194967 8518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:52.195747 8518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:52.196950 8518 hybrid_clock.cc:648] HybridClock initialized: now 1786333852196928 us; error 41 us; skew 500 ppm
Aug 10 03:50:52 dist-test-slave-pww0 krb5kdc[6667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333852, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:52.200246 8518 init.cc:377] Logged in from keytab as oryx/127.27.153.66@KRBTEST.COM (short username oryx)
I20260810 03:50:52.201287 8518 webserver.cc:460] Webserver started at http://127.27.153.66:46207/ using document root <none> and password file <none>
I20260810 03:50:52.201877 8518 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:52.201972 8518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:52.206370 8518 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:50:52.208910 8546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:52.210433 8518 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:50:52.210582 8518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "7a083bc0c0ba48e79563564743d2fb3f"
format_stamp: "Formatted at 2026-08-10 03:50:50 on dist-test-slave-pww0"
server_key: "2e43319a214928dd921d54318779336f"
server_key_iv: "8594e276226afea000f6b8ad9092f3e5"
server_key_version: "encryptionkey@0"
I20260810 03:50:52.211226 8518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:52.224898 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.221109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:39435 (local address 127.0.0.1:40170)
0810 03:50:52.221250 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:52.221267 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:52.221384 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:52.221655 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:52.221657 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:52.221672 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:52.221776 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.221780 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.222812 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.222815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.223655 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.223670 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.224554 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.224556 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.224626 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.224648 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 03:50:52.224694 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 03:50:52.224766 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 03:50:52.224980 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.221217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40170 (local address 127.27.153.124:39435)
0810 03:50:52.221357 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:52.221361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:52.221380 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:50:52.221470 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:52.221473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:52.221539 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:52.221640 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:52.221887 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.222658 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.223778 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.224441 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.224557 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.224748 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:52.224784 (+ 36us) server_negotiation.cc:299] Negotiation successful
0810 03:50:52.224858 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 03:50:52.225713 8221 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 03:50:52.226148 8277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3d2c2bbe823423ca1284d9fa2a52b2b. Latest consensus state: current_term: 1 leader_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: NON_VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: true } } }
I20260810 03:50:52.226286 8277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [sys.catalog]: This master's current role is: LEARNER
I20260810 03:50:52.229982 8518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:52.233244 8518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:52.233448 8518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:52.234092 8518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:52.235087 8518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:52.235148 8518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:52.235206 8518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:52.235232 8518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:52.244699 8518 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:41001
I20260810 03:50:52.244755 8738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:41001 every 8 connection(s)
I20260810 03:50:52.245808 8518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:50:52.249507 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 8518
I20260810 03:50:52.249701 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 7371
I20260810 03:50:52.266671 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:45049
--local_ip_for_outbound_sockets=127.27.153.67
--tserver_master_addrs=127.27.153.126:41933
--webserver_port=38001
--webserver_interface=127.27.153.67
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.67.keytab
--principal=oryx/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:39701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.153.126:41933,127.27.153.125:45527
--tserver_master_addrs=127.27.153.126:41933,127.27.153.125:45527,127.27.153.124:39435 with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:50:52 dist-test-slave-pww0 krb5kdc[6667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333852, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.125@KRBTEST.COM
Aug 10 03:50:52 dist-test-slave-pww0 krb5kdc[6667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333852, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM
Aug 10 03:50:52 dist-test-slave-pww0 krb5kdc[6667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333852, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.124@KRBTEST.COM
I20260810 03:50:52.275521 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.247772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:43465 (local address 127.27.153.126:41933)
0810 03:50:52.247929 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:52.247932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:52.259225 (+ 11293us) server_negotiation.cc:407] Connection header received
0810 03:50:52.260208 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:52.260212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:52.260267 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:52.260317 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:52.262685 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.263544 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.265627 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.265976 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.266054 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.270439 (+ 4385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:52.270444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:52.270447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:52.270481 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:52.272483 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.273280 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.273285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.273286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.273397 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.273978 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.273982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.273984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.274232 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:52.274309 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:52.275182 (+ 873us) server_negotiation.cc:299] Negotiation successful
0810 03:50:52.275318 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 03:50:52.275521 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.247777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:33665 (local address 127.27.153.124:39435)
0810 03:50:52.248016 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:52.248019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:52.249314 (+ 1295us) server_negotiation.cc:407] Connection header received
0810 03:50:52.260159 (+ 10845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:52.260164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:52.260231 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:52.260290 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:52.262680 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.263555 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.265621 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.265976 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.266054 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.270620 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:52.270624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:52.270627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:52.270660 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:52.272405 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.273280 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.273285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.273286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.273507 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.273978 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.273982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.273984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.274232 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:52.274309 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:52.275175 (+ 866us) server_negotiation.cc:299] Negotiation successful
0810 03:50:52.275318 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260810 03:50:52.275557 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.249792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:36781 (local address 127.27.153.125:45527)
0810 03:50:52.258737 (+ 8945us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:52.258742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:52.258768 (+ 26us) server_negotiation.cc:407] Connection header received
0810 03:50:52.260172 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:52.260176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:52.260238 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:52.260295 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:52.262669 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.263555 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.265611 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.265919 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.265998 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.271068 (+ 5070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:52.271072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:52.271074 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:52.271117 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:52.273256 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.273803 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.273807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.273809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.273909 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.274272 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.274275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.274277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.274532 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:52.274610 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:52.275203 (+ 593us) server_negotiation.cc:299] Negotiation successful
0810 03:50:52.275347 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8813}
I20260810 03:50:52.277386 8748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.249342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:45527 (local address 127.27.153.66:36781)
0810 03:50:52.249643 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:52.249657 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:52.259810 (+ 10153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:52.260817 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:52.260828 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:52.261577 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:52.262457 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.262480 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.263758 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.263762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.265450 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.265464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.266244 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.266248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.266534 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.267555 (+ 1021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:52.267570 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:52.270850 (+ 3280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:52.273407 (+ 2557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:52.273410 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:52.273413 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:52.273668 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:52.274037 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:52.274040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:52.274042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:52.274155 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:52.274693 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:52.274696 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:52.274868 (+ 172us) client_negotiation.cc:769] Sending connection context
0810 03:50:52.275727 (+ 859us) client_negotiation.cc:240] Negotiation successful
0810 03:50:52.276191 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":467,"spinlock_wait_cycles":123520,"thread_start_us":127,"threads_started":1}
I20260810 03:50:52.277387 8749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.256825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.66:43465)
0810 03:50:52.259088 (+ 2263us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:52.259107 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:52.259817 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:52.260817 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:52.260829 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:52.261600 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:52.262457 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.262480 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.263759 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.263762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.265456 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.265469 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.266244 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.266248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.266574 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.267377 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:52.267390 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:52.270140 (+ 2750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:52.272594 (+ 2454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:52.272597 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:52.272599 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:52.273145 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:52.273627 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:52.273629 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:52.273631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:52.273856 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:52.274398 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:52.274406 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:52.274863 (+ 457us) client_negotiation.cc:769] Sending connection context
0810 03:50:52.275722 (+ 859us) client_negotiation.cc:240] Negotiation successful
0810 03:50:52.276333 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2164,"mutex_wait_us":659,"spinlock_wait_cycles":40320,"thread_start_us":1789,"threads_started":1}
I20260810 03:50:52.277387 8747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.248293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:39435 (local address 127.27.153.66:33665)
0810 03:50:52.249088 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:52.249146 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:52.259866 (+ 10720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:52.260817 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:52.260828 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:52.261595 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:52.262459 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.262480 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.263759 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.263762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.265462 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.265468 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.266244 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.266248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.266569 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.267269 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:52.267291 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:52.270444 (+ 3153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:52.272562 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:52.272570 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:52.272591 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:52.273144 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:52.273599 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:52.273602 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:52.273604 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:52.273856 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:52.274398 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:52.274406 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:52.274863 (+ 457us) client_negotiation.cc:769] Sending connection context
0810 03:50:52.275888 (+ 1025us) client_negotiation.cc:240] Negotiation successful
0810 03:50:52.276263 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"mutex_wait_us":803,"spinlock_wait_cycles":36096,"thread_start_us":114,"threads_started":1}
I20260810 03:50:52.278847 8742 heartbeater.cc:347] Connected to a master server at 127.27.153.124:39435
I20260810 03:50:52.278836 8744 heartbeater.cc:347] Connected to a master server at 127.27.153.125:45527
I20260810 03:50:52.278837 8743 heartbeater.cc:347] Connected to a master server at 127.27.153.126:41933
I20260810 03:50:52.279145 8744 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:52.279211 8743 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:52.279146 8742 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:52.279870 8742 heartbeater.cc:511] Master 127.27.153.124:39435 requested a full tablet report, sending...
I20260810 03:50:52.279837 8744 heartbeater.cc:511] Master 127.27.153.125:45527 requested a full tablet report, sending...
I20260810 03:50:52.279838 8743 heartbeater.cc:511] Master 127.27.153.126:41933 requested a full tablet report, sending...
I20260810 03:50:52.281075 6760 ts_manager.cc:194] Re-registered known tserver with Master: 7a083bc0c0ba48e79563564743d2fb3f (127.27.153.66:41001)
I20260810 03:50:52.281149 7857 ts_manager.cc:194] Registered new tserver with Master: 7a083bc0c0ba48e79563564743d2fb3f (127.27.153.66:41001)
I20260810 03:50:52.281212 8191 ts_manager.cc:194] Registered new tserver with Master: 7a083bc0c0ba48e79563564743d2fb3f (127.27.153.66:41001)
I20260810 03:50:52.281870 6760 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:43465
W20260810 03:50:52.396509 8750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:52.396799 8750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:52.396847 8750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:52.401037 8750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:52.401113 8750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:52.401177 8750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:50:52.406723 8750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:39701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.67.keytab
--principal=oryx/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:45049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=38001
--tserver_master_addrs=127.27.153.126:41933,127.27.153.125:45527,127.27.153.124:39435
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:52.407918 8750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:52.409019 8750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:52.419809 8762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:52.419809 8759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:52.419817 8758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:52.420138 8750 server_base.cc:1034] running on GCE node
I20260810 03:50:52.420693 8750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:52.421461 8750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:52.422633 8750 hybrid_clock.cc:648] HybridClock initialized: now 1786333852422601 us; error 44 us; skew 500 ppm
Aug 10 03:50:52 dist-test-slave-pww0 krb5kdc[6667](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333852, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:52.425578 8750 init.cc:377] Logged in from keytab as oryx/127.27.153.67@KRBTEST.COM (short username oryx)
I20260810 03:50:52.426530 8750 webserver.cc:460] Webserver started at http://127.27.153.67:38001/ using document root <none> and password file <none>
I20260810 03:50:52.427150 8750 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:52.427232 8750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:52.431293 8750 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 03:50:52.433707 8779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:52.435319 8750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:50:52.435499 8750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "16dc897a7f744e78a9b0e59c3abd6d11"
format_stamp: "Formatted at 2026-08-10 03:50:51 on dist-test-slave-pww0"
server_key: "e27d02d96fd8c18dfe4499637c3e220d"
server_key_iv: "34cccd6bcf64359f57816673afa48805"
server_key_version: "encryptionkey@0"
I20260810 03:50:52.436146 8750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:52.455775 8750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:52.459206 8750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:52.459393 8750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:52.460045 8750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:50:52.460937 8750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:52.460984 8750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:52.461025 8750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:52.461046 8750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:52.471135 8750 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:45049
I20260810 03:50:52.471189 8981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:45049 every 8 connection(s)
I20260810 03:50:52.472285 8750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:50:52.476439 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 8750
Aug 10 03:50:52 dist-test-slave-pww0 krb5kdc[6667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333852, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM
Aug 10 03:50:52 dist-test-slave-pww0 krb5kdc[6667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333852, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.124@KRBTEST.COM
Aug 10 03:50:52 dist-test-slave-pww0 krb5kdc[6667](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333852, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.125@KRBTEST.COM
I20260810 03:50:52.490989 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.474602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:43621 (local address 127.27.153.126:41933)
0810 03:50:52.475050 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:52.475055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:52.476004 (+ 949us) server_negotiation.cc:407] Connection header received
0810 03:50:52.476705 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:52.476709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:52.476799 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:52.476875 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:52.478728 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.479608 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.481504 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.481848 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.481925 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.486740 (+ 4815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:52.486744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:52.486746 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:52.486774 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:52.488731 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.489355 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.489359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.489360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.489441 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.489739 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.489742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.489743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.489987 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:52.490056 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:52.490690 (+ 634us) server_negotiation.cc:299] Negotiation successful
0810 03:50:52.490826 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365}
I20260810 03:50:52.491011 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.474441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:44867 (local address 127.27.153.125:45527)
0810 03:50:52.474895 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:52.474900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:52.475538 (+ 638us) server_negotiation.cc:407] Connection header received
0810 03:50:52.476724 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:52.476727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:52.476788 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:52.476845 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:52.478740 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.479637 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.481531 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.481894 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.481959 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.486585 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:52.486590 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:52.486592 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:52.486621 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:52.488357 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.489023 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.489028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.489030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.489119 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.489552 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.489555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.489557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.489800 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:52.489872 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:52.490690 (+ 818us) server_negotiation.cc:299] Negotiation successful
0810 03:50:52.490836 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":355}
I20260810 03:50:52.493287 8989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.474769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:45527 (local address 127.27.153.67:44867)
0810 03:50:52.475341 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:52.475397 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:52.476451 (+ 1054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:52.477073 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:52.477081 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:52.477669 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:52.478547 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.478568 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.479784 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.479787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.481412 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.481420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.482142 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.482146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.482475 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.483237 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:52.483256 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:52.486430 (+ 3174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:52.488501 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:52.488508 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:52.488532 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:52.488921 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:52.489235 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:52.489237 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:52.489239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:52.489441 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:52.489959 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:52.489967 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:52.490378 (+ 411us) client_negotiation.cc:769] Sending connection context
0810 03:50:52.490810 (+ 432us) client_negotiation.cc:240] Negotiation successful
0810 03:50:52.492008 (+ 1198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":343,"spinlock_wait_cycles":11136,"thread_start_us":153,"threads_started":1}
I20260810 03:50:52.494076 9001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.479820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47696 (local address 127.27.153.125:45527)
0810 03:50:52.481341 (+ 1521us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:52.481346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:52.481359 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:50:52.481396 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:52.481398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:52.481447 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:52.481522 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:52.483370 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.484178 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.485933 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.486255 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.486356 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.488034 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:52.488038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:52.488040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:52.488069 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:52.491574 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.492152 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.492155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.492157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.492249 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.492528 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.492531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.492533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.492772 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:52.492892 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:52.493766 (+ 874us) server_negotiation.cc:299] Negotiation successful
0810 03:50:52.493906 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1693,"server-negotiator.queue_time_us":1440,"thread_start_us":92,"threads_started":1}
I20260810 03:50:52.493289 8990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.474769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.67:43621)
0810 03:50:52.475905 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:52.475917 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:52.476457 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:52.477073 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:52.477080 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:52.477669 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:52.478547 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.478568 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.480093 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.480096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.481359 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.481370 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.482142 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.482146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.482478 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.483195 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:52.483236 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:52.486555 (+ 3319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:52.488939 (+ 2384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:52.488943 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:52.488945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:52.489245 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:52.489552 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:52.489554 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:52.489556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:52.489637 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:52.490191 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:52.490194 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:52.490377 (+ 183us) client_negotiation.cc:769] Sending connection context
0810 03:50:52.491237 (+ 860us) client_negotiation.cc:240] Negotiation successful
0810 03:50:52.492769 (+ 1532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1049,"mutex_wait_us":1732,"spinlock_wait_cycles":14080,"thread_start_us":152,"threads_started":1}
I20260810 03:50:52.494764 8987 heartbeater.cc:347] Connected to a master server at 127.27.153.125:45527
I20260810 03:50:52.495021 8987 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:52.495082 8986 heartbeater.cc:347] Connected to a master server at 127.27.153.126:41933
I20260810 03:50:52.495170 8986 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:52.495671 8987 heartbeater.cc:511] Master 127.27.153.125:45527 requested a full tablet report, sending...
I20260810 03:50:52.495672 8986 heartbeater.cc:511] Master 127.27.153.126:41933 requested a full tablet report, sending...
I20260810 03:50:52.496325 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.474743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:60667 (local address 127.27.153.124:39435)
0810 03:50:52.474940 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:52.474944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:52.477458 (+ 2514us) server_negotiation.cc:407] Connection header received
0810 03:50:52.477672 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:52.477675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:52.477732 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:52.477787 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:52.478751 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.479546 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.481503 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.481824 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.481914 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.486987 (+ 5073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:52.486991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:52.486994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:52.487025 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:52.489851 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.494863 (+ 5012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.494867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.494869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.494979 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.495337 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.495341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.495343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.495596 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:52.495685 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:52.496032 (+ 347us) server_negotiation.cc:299] Negotiation successful
0810 03:50:52.496179 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260810 03:50:52.496809 8991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.474769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:39435 (local address 127.27.153.67:60667)
0810 03:50:52.477357 (+ 2588us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:52.477372 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:52.477537 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:52.477932 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:52.477935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:52.478124 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:52.478581 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.478593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.479739 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.479742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.481354 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.481365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.482142 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.482146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.482437 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.483288 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:52.483303 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:52.486015 (+ 2712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:52.491273 (+ 5258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:52.491276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:52.491278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:52.491839 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:52.495100 (+ 3261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:52.495103 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:52.495116 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:52.495214 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:52.495746 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:52.495748 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:52.495839 (+ 91us) client_negotiation.cc:769] Sending connection context
0810 03:50:52.496467 (+ 628us) client_negotiation.cc:240] Negotiation successful
0810 03:50:52.496635 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2484,"mutex_wait_us":526,"thread_start_us":106,"threads_started":1}
I20260810 03:50:52.496968 7857 ts_manager.cc:194] Registered new tserver with Master: 16dc897a7f744e78a9b0e59c3abd6d11 (127.27.153.67:45049)
I20260810 03:50:52.497165 6760 ts_manager.cc:194] Re-registered known tserver with Master: 16dc897a7f744e78a9b0e59c3abd6d11 (127.27.153.67:45049)
I20260810 03:50:52.497782 6760 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:43621
I20260810 03:50:52.497800 8982 heartbeater.cc:347] Connected to a master server at 127.27.153.124:39435
I20260810 03:50:52.497881 8982 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:52.498078 8982 heartbeater.cc:511] Master 127.27.153.124:39435 requested a full tablet report, sending...
I20260810 03:50:52.498797 8191 ts_manager.cc:194] Registered new tserver with Master: 16dc897a7f744e78a9b0e59c3abd6d11 (127.27.153.67:45049)
I20260810 03:50:52.500417 8998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.478896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40186 (local address 127.27.153.124:39435)
0810 03:50:52.482589 (+ 3693us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:52.482593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:52.484597 (+ 2004us) server_negotiation.cc:407] Connection header received
0810 03:50:52.484663 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:52.484666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:52.484726 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:52.484809 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:52.485840 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.486647 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.490272 (+ 3625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.490801 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.490917 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.492803 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:52.492807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:52.492810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:52.492840 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:52.498492 (+ 5652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.498991 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.498995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.498997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.499094 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.499425 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.499428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.499430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.499699 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:52.499841 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:52.500113 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 03:50:52.500249 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3614,"thread_start_us":63,"threads_started":1}
I20260810 03:50:52.502621 9002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.479862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48354 (local address 127.27.153.126:41933)
0810 03:50:52.488107 (+ 8245us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:52.488112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:52.488127 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:50:52.488182 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:52.488185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:52.488249 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:52.488344 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:52.491796 (+ 3452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.492592 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.495556 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.495892 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.495997 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.499515 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:52.499519 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:52.499522 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:52.499550 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:52.501263 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.501671 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.501673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.501674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.501757 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.501955 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.501957 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.501958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.502121 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:52.502209 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:52.502364 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 03:50:52.502469 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8145,"thread_start_us":96,"threads_started":1}
I20260810 03:50:52.505532 6760 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:50:52.508739 6760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:50:52.511540 8143 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: NON_VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: true } } }
I20260810 03:50:52.511708 8143 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:52.515156 8143 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b: attempting to promote NON_VOTER 0fed699bb95348be8b9a62282240a7de to VOTER
I20260810 03:50:52.515525 8143 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: false } }
I20260810 03:50:52.516763 7875 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 1 FOLLOWER]: Refusing update from remote peer f3d2c2bbe823423ca1284d9fa2a52b2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 03:50:52.518250 9011 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 03:50:52.520809 9011 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: NON_VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: true } } } pending_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: false } } }
I20260810 03:50:52.520980 9011 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:52.525635 8221 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 1 LEARNER]: Refusing update from remote peer f3d2c2bbe823423ca1284d9fa2a52b2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 03:50:52.526007 9014 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0fed699bb95348be8b9a62282240a7de (127.27.153.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: false } } }
I20260810 03:50:52.526216 9015 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 03:50:52.527664 9014 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: false } } }
I20260810 03:50:52.527940 9014 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:52.527992 8221 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0fed699bb95348be8b9a62282240a7de (127.27.153.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: false } } }
I20260810 03:50:52.529397 7875 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 0fed699bb95348be8b9a62282240a7de (127.27.153.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: false } } }
I20260810 03:50:52.530086 8277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: false } } }
I20260810 03:50:52.530226 8277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:50:52.531383 9027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: false } } }
I20260810 03:50:52.531570 9027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:50:52.538327 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.531755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:41001 (local address 127.0.0.1:33474)
0810 03:50:52.532163 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:52.532180 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:52.532290 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:52.533097 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:52.533101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:52.533124 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:52.533248 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.533256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.536021 (+ 2765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.536025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.536859 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.536870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.537902 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.537906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.537997 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.538025 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 03:50:52.538086 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 03:50:52.538159 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319}
I20260810 03:50:52.538408 9030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.532120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33474 (local address 127.27.153.66:41001)
0810 03:50:52.532535 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:52.532549 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:52.532579 (+ 30us) server_negotiation.cc:407] Connection header received
0810 03:50:52.532698 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:52.532704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:52.532915 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:52.534412 (+ 1497us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:52.534576 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.535914 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.536983 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.537783 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.537901 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.538073 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:52.538132 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 03:50:52.538251 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":180,"threads_started":1}
I20260810 03:50:52.538491 9033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.533127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:45049 (local address 127.0.0.1:47746)
0810 03:50:52.533422 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:52.533434 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:52.533514 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:52.534172 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:52.534174 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:52.534194 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:52.534304 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.534311 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.535904 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.535908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.537080 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.537091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.538135 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.538139 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.538229 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.538253 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:50:52.538304 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 03:50:52.538372 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":137,"threads_started":1}
I20260810 03:50:52.539217 9035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.533446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47746 (local address 127.27.153.67:45049)
0810 03:50:52.533768 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:52.533778 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:52.533798 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:50:52.533862 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:52.533866 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:52.534035 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:52.534187 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:52.534456 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.535743 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.537227 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.537996 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.538744 (+ 748us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.538897 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:52.538949 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 03:50:52.539054 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":121,"threads_started":1}
I20260810 03:50:52.540256 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.533531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:35535 (local address 127.0.0.1:40224)
0810 03:50:52.534203 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:52.534215 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:52.534306 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:52.534953 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:52.534956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:52.534976 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:52.535074 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.535080 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.536529 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.536532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.537585 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.537596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.539842 (+ 2246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.539846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.539940 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.539965 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 03:50:52.540034 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 03:50:52.540120 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":82,"threads_started":1}
I20260810 03:50:52.540865 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.533857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40224 (local address 127.27.153.65:35535)
0810 03:50:52.534101 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:52.534117 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:52.534295 (+ 178us) server_negotiation.cc:407] Connection header received
0810 03:50:52.534467 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:52.534472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:52.534629 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:52.534792 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:52.535213 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.536395 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.537730 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.539485 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.539597 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.539774 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:52.540590 (+ 816us) server_negotiation.cc:299] Negotiation successful
0810 03:50:52.540703 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":70,"threads_started":1}
I20260810 03:50:52.542011 8627 tablet_service.cc:1467] Processing CreateTablet for tablet 73cf7ae3f67546b1bb33ed451c54fda1 (DEFAULT_TABLE table=test-table [id=36e6a99dc6674869b18f10f0ce2a1dc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:52.542033 8862 tablet_service.cc:1467] Processing CreateTablet for tablet 73cf7ae3f67546b1bb33ed451c54fda1 (DEFAULT_TABLE table=test-table [id=36e6a99dc6674869b18f10f0ce2a1dc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:52.543133 8627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73cf7ae3f67546b1bb33ed451c54fda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:52.543431 8862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73cf7ae3f67546b1bb33ed451c54fda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:52.544548 8389 tablet_service.cc:1467] Processing CreateTablet for tablet 73cf7ae3f67546b1bb33ed451c54fda1 (DEFAULT_TABLE table=test-table [id=36e6a99dc6674869b18f10f0ce2a1dc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:52.545713 8389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73cf7ae3f67546b1bb33ed451c54fda1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:52.551141 9039 tablet_bootstrap.cc:492] T 73cf7ae3f67546b1bb33ed451c54fda1 P 7a083bc0c0ba48e79563564743d2fb3f: Bootstrap starting.
I20260810 03:50:52.552445 9040 tablet_bootstrap.cc:492] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11: Bootstrap starting.
I20260810 03:50:52.553817 9039 tablet_bootstrap.cc:654] T 73cf7ae3f67546b1bb33ed451c54fda1 P 7a083bc0c0ba48e79563564743d2fb3f: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:52.554742 9040 tablet_bootstrap.cc:654] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:52.554787 9039 log.cc:824] T 73cf7ae3f67546b1bb33ed451c54fda1 P 7a083bc0c0ba48e79563564743d2fb3f: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:52.555634 9040 log.cc:824] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:52.557319 9039 tablet_bootstrap.cc:492] T 73cf7ae3f67546b1bb33ed451c54fda1 P 7a083bc0c0ba48e79563564743d2fb3f: No bootstrap required, opened a new log
I20260810 03:50:52.557526 9039 ts_tablet_manager.cc:1397] T 73cf7ae3f67546b1bb33ed451c54fda1 P 7a083bc0c0ba48e79563564743d2fb3f: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 03:50:52.558022 9040 tablet_bootstrap.cc:492] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11: No bootstrap required, opened a new log
I20260810 03:50:52.558220 9040 ts_tablet_manager.cc:1397] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:50:52.560268 9047 tablet_bootstrap.cc:492] T 73cf7ae3f67546b1bb33ed451c54fda1 P 6720a63653bd424bba05965f0b0844dc: Bootstrap starting.
I20260810 03:50:52.562971 9040 raft_consensus.cc:348] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a083bc0c0ba48e79563564743d2fb3f" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 41001 } } peers { permanent_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 45049 } } peers { permanent_uuid: "6720a63653bd424bba05965f0b0844dc" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 35535 } }
I20260810 03:50:52.563313 9040 raft_consensus.cc:374] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:52.563387 9040 raft_consensus.cc:729] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16dc897a7f744e78a9b0e59c3abd6d11, State: Initialized, Role: FOLLOWER
I20260810 03:50:52.562973 9039 raft_consensus.cc:348] T 73cf7ae3f67546b1bb33ed451c54fda1 P 7a083bc0c0ba48e79563564743d2fb3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a083bc0c0ba48e79563564743d2fb3f" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 41001 } } peers { permanent_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 45049 } } peers { permanent_uuid: "6720a63653bd424bba05965f0b0844dc" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 35535 } }
I20260810 03:50:52.563720 9039 raft_consensus.cc:374] T 73cf7ae3f67546b1bb33ed451c54fda1 P 7a083bc0c0ba48e79563564743d2fb3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:52.564007 9039 raft_consensus.cc:729] T 73cf7ae3f67546b1bb33ed451c54fda1 P 7a083bc0c0ba48e79563564743d2fb3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a083bc0c0ba48e79563564743d2fb3f, State: Initialized, Role: FOLLOWER
I20260810 03:50:52.564028 9047 tablet_bootstrap.cc:654] T 73cf7ae3f67546b1bb33ed451c54fda1 P 6720a63653bd424bba05965f0b0844dc: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:52.563977 9040 consensus_queue.cc:260] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a083bc0c0ba48e79563564743d2fb3f" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 41001 } } peers { permanent_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 45049 } } peers { permanent_uuid: "6720a63653bd424bba05965f0b0844dc" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 35535 } }
I20260810 03:50:52.564774 9039 consensus_queue.cc:260] T 73cf7ae3f67546b1bb33ed451c54fda1 P 7a083bc0c0ba48e79563564743d2fb3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a083bc0c0ba48e79563564743d2fb3f" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 41001 } } peers { permanent_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 45049 } } peers { permanent_uuid: "6720a63653bd424bba05965f0b0844dc" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 35535 } }
I20260810 03:50:52.565128 9047 log.cc:824] T 73cf7ae3f67546b1bb33ed451c54fda1 P 6720a63653bd424bba05965f0b0844dc: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:52.565289 9040 ts_tablet_manager.cc:1428] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 03:50:52.565867 8986 heartbeater.cc:503] Master 127.27.153.126:41933 was elected leader, sending a full tablet report...
I20260810 03:50:52.567659 9039 ts_tablet_manager.cc:1428] T 73cf7ae3f67546b1bb33ed451c54fda1 P 7a083bc0c0ba48e79563564743d2fb3f: Time spent starting tablet: real 0.010s user 0.001s sys 0.006s
I20260810 03:50:52.567945 8743 heartbeater.cc:503] Master 127.27.153.126:41933 was elected leader, sending a full tablet report...
I20260810 03:50:52.567946 9047 tablet_bootstrap.cc:492] T 73cf7ae3f67546b1bb33ed451c54fda1 P 6720a63653bd424bba05965f0b0844dc: No bootstrap required, opened a new log
I20260810 03:50:52.568368 9047 ts_tablet_manager.cc:1397] T 73cf7ae3f67546b1bb33ed451c54fda1 P 6720a63653bd424bba05965f0b0844dc: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 03:50:52.573205 9047 raft_consensus.cc:348] T 73cf7ae3f67546b1bb33ed451c54fda1 P 6720a63653bd424bba05965f0b0844dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a083bc0c0ba48e79563564743d2fb3f" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 41001 } } peers { permanent_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 45049 } } peers { permanent_uuid: "6720a63653bd424bba05965f0b0844dc" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 35535 } }
I20260810 03:50:52.573585 9047 raft_consensus.cc:374] T 73cf7ae3f67546b1bb33ed451c54fda1 P 6720a63653bd424bba05965f0b0844dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:52.573647 9047 raft_consensus.cc:729] T 73cf7ae3f67546b1bb33ed451c54fda1 P 6720a63653bd424bba05965f0b0844dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6720a63653bd424bba05965f0b0844dc, State: Initialized, Role: FOLLOWER
I20260810 03:50:52.574210 9047 consensus_queue.cc:260] T 73cf7ae3f67546b1bb33ed451c54fda1 P 6720a63653bd424bba05965f0b0844dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a083bc0c0ba48e79563564743d2fb3f" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 41001 } } peers { permanent_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 45049 } } peers { permanent_uuid: "6720a63653bd424bba05965f0b0844dc" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 35535 } }
I20260810 03:50:52.575445 9047 ts_tablet_manager.cc:1428] T 73cf7ae3f67546b1bb33ed451c54fda1 P 6720a63653bd424bba05965f0b0844dc: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 03:50:52.576103 8508 heartbeater.cc:503] Master 127.27.153.126:41933 was elected leader, sending a full tablet report...
W20260810 03:50:52.723517 8988 tablet.cc:2369] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:50:52.747742 8746 tablet.cc:2369] T 73cf7ae3f67546b1bb33ed451c54fda1 P 7a083bc0c0ba48e79563564743d2fb3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:50:52.798696 8511 tablet.cc:2369] T 73cf7ae3f67546b1bb33ed451c54fda1 P 6720a63653bd424bba05965f0b0844dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:50:52.816746 9048 raft_consensus.cc:482] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:50:52.816956 9048 raft_consensus.cc:504] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a083bc0c0ba48e79563564743d2fb3f" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 41001 } } peers { permanent_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 45049 } } peers { permanent_uuid: "6720a63653bd424bba05965f0b0844dc" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 35535 } }
I20260810 03:50:52.818151 9048 leader_election.cc:288] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a083bc0c0ba48e79563564743d2fb3f (127.27.153.66:41001), 6720a63653bd424bba05965f0b0844dc (127.27.153.65:35535)
I20260810 03:50:52.821959 9030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.818542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:41693 (local address 127.27.153.66:41001)
0810 03:50:52.818689 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:52.818693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:52.818708 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:50:52.818862 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:52.818865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:52.818934 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:52.818998 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:52.819294 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.819808 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.820913 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.821397 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.821509 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.821567 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:52.821753 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 03:50:52.821823 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 03:50:52.821971 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.818529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:40749 (local address 127.27.153.65:35535)
0810 03:50:52.818659 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:52.818664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:52.818724 (+ 60us) server_negotiation.cc:407] Connection header received
0810 03:50:52.818875 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:52.818878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:52.818930 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:52.818995 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:52.819255 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.819827 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.820971 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.821443 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.821551 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.821603 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:52.821784 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 03:50:52.821857 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:50:52.821966 8989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.818436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:41001 (local address 127.27.153.67:41693)
0810 03:50:52.818643 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:52.818658 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:52.818757 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:52.819019 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:52.819022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:52.819066 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:52.819188 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.819194 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.819910 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.819913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.820793 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.820803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.821509 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.821512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.821613 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.821665 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 03:50:52.821728 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 03:50:52.821818 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":23,"spinlock_wait_cycles":1792}
I20260810 03:50:52.821974 8991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.818484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:35535 (local address 127.27.153.67:40749)
0810 03:50:52.818657 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:52.818670 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:52.818777 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:52.819015 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:52.819017 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:52.819033 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:52.819154 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.819159 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.819935 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.819938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.820854 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:52.820864 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.821551 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.821554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:52.821667 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.821698 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 03:50:52.821760 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 03:50:52.821831 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":1,"spinlock_wait_cycles":2176}
I20260810 03:50:52.822602 8428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cf7ae3f67546b1bb33ed451c54fda1" candidate_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6720a63653bd424bba05965f0b0844dc" is_pre_election: true
I20260810 03:50:52.822717 8660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cf7ae3f67546b1bb33ed451c54fda1" candidate_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a083bc0c0ba48e79563564743d2fb3f" is_pre_election: true
I20260810 03:50:52.822933 8428 raft_consensus.cc:2454] T 73cf7ae3f67546b1bb33ed451c54fda1 P 6720a63653bd424bba05965f0b0844dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16dc897a7f744e78a9b0e59c3abd6d11 in term 0.
I20260810 03:50:52.823010 8660 raft_consensus.cc:2454] T 73cf7ae3f67546b1bb33ed451c54fda1 P 7a083bc0c0ba48e79563564743d2fb3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16dc897a7f744e78a9b0e59c3abd6d11 in term 0.
I20260810 03:50:52.823393 8780 leader_election.cc:302] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 16dc897a7f744e78a9b0e59c3abd6d11, 6720a63653bd424bba05965f0b0844dc, 7a083bc0c0ba48e79563564743d2fb3f; no voters:
I20260810 03:50:52.823704 9048 raft_consensus.cc:2781] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:50:52.823786 9048 raft_consensus.cc:482] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:50:52.823833 9048 raft_consensus.cc:3037] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:52.824889 9048 raft_consensus.cc:504] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a083bc0c0ba48e79563564743d2fb3f" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 41001 } } peers { permanent_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 45049 } } peers { permanent_uuid: "6720a63653bd424bba05965f0b0844dc" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 35535 } }
I20260810 03:50:52.825258 9048 leader_election.cc:288] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11 [CANDIDATE]: Term 1 election: Requested vote from peers 7a083bc0c0ba48e79563564743d2fb3f (127.27.153.66:41001), 6720a63653bd424bba05965f0b0844dc (127.27.153.65:35535)
I20260810 03:50:52.825625 8660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cf7ae3f67546b1bb33ed451c54fda1" candidate_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a083bc0c0ba48e79563564743d2fb3f"
I20260810 03:50:52.825667 8428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73cf7ae3f67546b1bb33ed451c54fda1" candidate_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6720a63653bd424bba05965f0b0844dc"
I20260810 03:50:52.825762 8660 raft_consensus.cc:3037] T 73cf7ae3f67546b1bb33ed451c54fda1 P 7a083bc0c0ba48e79563564743d2fb3f [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:52.825780 8428 raft_consensus.cc:3037] T 73cf7ae3f67546b1bb33ed451c54fda1 P 6720a63653bd424bba05965f0b0844dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:52.827021 8428 raft_consensus.cc:2454] T 73cf7ae3f67546b1bb33ed451c54fda1 P 6720a63653bd424bba05965f0b0844dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16dc897a7f744e78a9b0e59c3abd6d11 in term 1.
I20260810 03:50:52.827021 8660 raft_consensus.cc:2454] T 73cf7ae3f67546b1bb33ed451c54fda1 P 7a083bc0c0ba48e79563564743d2fb3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16dc897a7f744e78a9b0e59c3abd6d11 in term 1.
I20260810 03:50:52.827335 8783 leader_election.cc:302] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16dc897a7f744e78a9b0e59c3abd6d11, 7a083bc0c0ba48e79563564743d2fb3f; no voters:
I20260810 03:50:52.827556 9048 raft_consensus.cc:2781] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:52.827811 9048 raft_consensus.cc:686] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11 [term 1 LEADER]: Becoming Leader. State: Replica: 16dc897a7f744e78a9b0e59c3abd6d11, State: Running, Role: LEADER
I20260810 03:50:52.828214 9048 consensus_queue.cc:237] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a083bc0c0ba48e79563564743d2fb3f" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 41001 } } peers { permanent_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 45049 } } peers { permanent_uuid: "6720a63653bd424bba05965f0b0844dc" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 35535 } }
I20260810 03:50:52.832015 6760 catalog_manager.cc:5495] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16dc897a7f744e78a9b0e59c3abd6d11 (127.27.153.67). New cstate: current_term: 1 leader_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a083bc0c0ba48e79563564743d2fb3f" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 41001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 45049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6720a63653bd424bba05965f0b0844dc" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 35535 } health_report { overall_health: UNKNOWN } } }
I20260810 03:50:52.833685 9015 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: false } } }
I20260810 03:50:52.833837 9015 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:52.907570 9035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.903979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47762 (local address 127.27.153.67:45049)
0810 03:50:52.904121 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:52.904126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:52.904160 (+ 34us) server_negotiation.cc:407] Connection header received
0810 03:50:52.904253 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:52.904255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:52.904313 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:52.904407 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:50:52.904663 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.905193 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.906045 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.906266 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.906359 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.906679 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:50:52.907110 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:50:52.907232 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:52.907302 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 03:50:52.907370 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:50:52.913702 8660 raft_consensus.cc:1261] T 73cf7ae3f67546b1bb33ed451c54fda1 P 7a083bc0c0ba48e79563564743d2fb3f [term 1 FOLLOWER]: Refusing update from remote peer 16dc897a7f744e78a9b0e59c3abd6d11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:52.913702 8428 raft_consensus.cc:1261] T 73cf7ae3f67546b1bb33ed451c54fda1 P 6720a63653bd424bba05965f0b0844dc [term 1 FOLLOWER]: Refusing update from remote peer 16dc897a7f744e78a9b0e59c3abd6d11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:52.914439 9048 consensus_queue.cc:1035] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6720a63653bd424bba05965f0b0844dc" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 35535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:52.914654 9058 consensus_queue.cc:1035] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a083bc0c0ba48e79563564743d2fb3f" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 41001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:52.925989 9065 mvcc.cc:205] Tried to move back new op lower bound from 7316823461526929408 to 7316823461187072000. Current Snapshot: MvccSnapshot[applied={T|T < 7316823461526929408}]
I20260810 03:50:52.926656 9069 mvcc.cc:205] Tried to move back new op lower bound from 7316823461526929408 to 7316823461187072000. Current Snapshot: MvccSnapshot[applied={T|T < 7316823461526929408}]
I20260810 03:50:52.930347 9070 mvcc.cc:205] Tried to move back new op lower bound from 7316823461526929408 to 7316823461187072000. Current Snapshot: MvccSnapshot[applied={T|T < 7316823461526929408}]
I20260810 03:50:52.938450 6754 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:52.938628 6754 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:52.944187 6754 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3d2c2bbe823423ca1284d9fa2a52b2b: Sending DeleteTablet for 3 replicas of tablet 73cf7ae3f67546b1bb33ed451c54fda1
I20260810 03:50:52.945248 8389 tablet_service.cc:1514] Processing DeleteTablet for tablet 73cf7ae3f67546b1bb33ed451c54fda1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:52 UTC) from {username='oryx', principal='oryx/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:40224
I20260810 03:50:52.945298 8627 tablet_service.cc:1514] Processing DeleteTablet for tablet 73cf7ae3f67546b1bb33ed451c54fda1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:52 UTC) from {username='oryx', principal='oryx/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:33474
I20260810 03:50:52.946031 9086 tablet_replica.cc:330] T 73cf7ae3f67546b1bb33ed451c54fda1 P 7a083bc0c0ba48e79563564743d2fb3f: stopping tablet replica
I20260810 03:50:52.946086 8862 tablet_service.cc:1514] Processing DeleteTablet for tablet 73cf7ae3f67546b1bb33ed451c54fda1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:52 UTC) from {username='oryx', principal='oryx/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:47746
I20260810 03:50:52.946406 9086 raft_consensus.cc:2229] T 73cf7ae3f67546b1bb33ed451c54fda1 P 7a083bc0c0ba48e79563564743d2fb3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:52.946776 9086 raft_consensus.cc:2258] T 73cf7ae3f67546b1bb33ed451c54fda1 P 7a083bc0c0ba48e79563564743d2fb3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:52.947208 9088 tablet_replica.cc:330] T 73cf7ae3f67546b1bb33ed451c54fda1 P 6720a63653bd424bba05965f0b0844dc: stopping tablet replica
I20260810 03:50:52.947575 9088 raft_consensus.cc:2229] T 73cf7ae3f67546b1bb33ed451c54fda1 P 6720a63653bd424bba05965f0b0844dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:52.947894 9088 raft_consensus.cc:2258] T 73cf7ae3f67546b1bb33ed451c54fda1 P 6720a63653bd424bba05965f0b0844dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:52.948792 9086 ts_tablet_manager.cc:1905] T 73cf7ae3f67546b1bb33ed451c54fda1 P 7a083bc0c0ba48e79563564743d2fb3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:52.948998 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 6697
I20260810 03:50:52.958295 9088 ts_tablet_manager.cc:1905] T 73cf7ae3f67546b1bb33ed451c54fda1 P 6720a63653bd424bba05965f0b0844dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:52.961459 9088 ts_tablet_manager.cc:1918] T 73cf7ae3f67546b1bb33ed451c54fda1 P 6720a63653bd424bba05965f0b0844dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:50:52.961601 9088 log.cc:1196] T 73cf7ae3f67546b1bb33ed451c54fda1 P 6720a63653bd424bba05965f0b0844dc: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-0/wal/wals/73cf7ae3f67546b1bb33ed451c54fda1
I20260810 03:50:52.961825 9086 ts_tablet_manager.cc:1918] T 73cf7ae3f67546b1bb33ed451c54fda1 P 7a083bc0c0ba48e79563564743d2fb3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:50:52.962039 9088 ts_tablet_manager.cc:1939] T 73cf7ae3f67546b1bb33ed451c54fda1 P 6720a63653bd424bba05965f0b0844dc: Deleting consensus metadata
I20260810 03:50:52.962042 9086 log.cc:1196] T 73cf7ae3f67546b1bb33ed451c54fda1 P 7a083bc0c0ba48e79563564743d2fb3f: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/73cf7ae3f67546b1bb33ed451c54fda1
I20260810 03:50:52.962515 9086 ts_tablet_manager.cc:1939] T 73cf7ae3f67546b1bb33ed451c54fda1 P 7a083bc0c0ba48e79563564743d2fb3f: Deleting consensus metadata
I20260810 03:50:52.967813 9090 tablet_replica.cc:330] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11: stopping tablet replica
I20260810 03:50:52.968366 9090 raft_consensus.cc:2229] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:50:52.968987 9090 raft_consensus.cc:2258] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 03:50:52.970063 8313 connection.cc:535] server connection from 127.0.0.1:40224 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 03:50:52.970794 8553 connection.cc:535] server connection from 127.0.0.1:33474 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 03:50:52.971141 9090 ts_tablet_manager.cc:1905] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:52.974269 9090 ts_tablet_manager.cc:1918] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:50:52.974458 9090 log.cc:1196] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/73cf7ae3f67546b1bb33ed451c54fda1
I20260810 03:50:52.974872 9090 ts_tablet_manager.cc:1939] T 73cf7ae3f67546b1bb33ed451c54fda1 P 16dc897a7f744e78a9b0e59c3abd6d11: Deleting consensus metadata
W20260810 03:50:52.976588 8780 connection.cc:441] server connection from 127.0.0.1:47746 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47746 (request call id 1) could send its response
I20260810 03:50:52.978555 9001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.946930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47700 (local address 127.27.153.125:45527)
0810 03:50:52.947338 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:52.947343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:52.968173 (+ 20830us) server_negotiation.cc:407] Connection header received
0810 03:50:52.968410 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:52.968413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:52.968469 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:52.968523 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:52.969603 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.970516 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.971825 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.972200 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.972277 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.974838 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:52.974843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:52.974846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:52.974879 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:52.976669 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.977169 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.977173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.977175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.977265 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.977562 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.977565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.977567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.977797 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:52.977899 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:52.978251 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 03:50:52.978390 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314}
I20260810 03:50:52.985805 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.975390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47714 (local address 127.27.153.125:45527)
0810 03:50:52.975530 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:52.975534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:52.976732 (+ 1198us) server_negotiation.cc:407] Connection header received
0810 03:50:52.976933 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:52.976936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:52.976989 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:52.977038 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:52.978005 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.978778 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.979963 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.980286 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.980359 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.982042 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:52.982046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:52.982047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:52.982071 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:52.983785 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.984303 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.984307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.984309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.984411 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.984738 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.984742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.984744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.984985 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:52.985105 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:52.985488 (+ 383us) server_negotiation.cc:299] Negotiation successful
0810 03:50:52.985626 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 03:50:52.986830 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.976110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40212 (local address 127.27.153.124:39435)
0810 03:50:52.976268 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:52.976273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:52.977496 (+ 1223us) server_negotiation.cc:407] Connection header received
0810 03:50:52.977719 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:52.977722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:52.977776 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:52.977853 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:52.979059 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.979882 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.980965 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.981266 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.981371 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.983266 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:52.983271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:52.983273 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:52.983307 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:52.985023 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.985479 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.985483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.985484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.985570 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.985895 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.985898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.985900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.986127 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:52.986226 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:52.986561 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 03:50:52.986670 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:50:52.989203 8998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:52.947698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.27.153.124:39435)
0810 03:50:52.947828 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:52.947832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:52.980048 (+ 32216us) server_negotiation.cc:407] Connection header received
0810 03:50:52.980211 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:52.980215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:52.980268 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:52.980316 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:52.981486 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.982593 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.983784 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:52.984179 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:52.984257 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:52.985828 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:52.985832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:52.985835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:52.985861 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:52.987520 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.987999 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.988002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.988004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.988087 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:52.988366 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:52.988369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:52.988371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:52.988580 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:52.988698 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:52.988932 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 03:50:52.989038 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:50:53.927726 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:53.927199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.65:51929)
0810 03:50:53.927488 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:53.927622 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":83,"threads_started":1}
W20260810 03:50:53.928092 8508 heartbeater.cc:644] Failed to heartbeat to 127.27.153.126:41933 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
I20260810 03:50:53.933476 9108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:53.932927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.66:54955)
0810 03:50:53.933251 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:53.933359 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":80,"threads_started":1}
I20260810 03:50:53.933642 9109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:53.933244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.67:35489)
0810 03:50:53.933468 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:53.933543 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":84,"threads_started":1}
W20260810 03:50:53.933809 8743 heartbeater.cc:644] Failed to heartbeat to 127.27.153.126:41933 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
W20260810 03:50:53.933893 8986 heartbeater.cc:644] Failed to heartbeat to 127.27.153.126:41933 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
I20260810 03:50:54.444139 9111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f3d2c2bbe823423ca1284d9fa2a52b2b)
I20260810 03:50:54.444156 9110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f3d2c2bbe823423ca1284d9fa2a52b2b)
I20260810 03:50:54.444367 9110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: false } }
I20260810 03:50:54.444388 9111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: false } }
I20260810 03:50:54.445369 9110 leader_election.cc:288] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3d2c2bbe823423ca1284d9fa2a52b2b (127.27.153.126:41933), 0fed699bb95348be8b9a62282240a7de (127.27.153.124:39435)
I20260810 03:50:54.445353 9111 leader_election.cc:288] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3d2c2bbe823423ca1284d9fa2a52b2b (127.27.153.126:41933), 5e655481b21e4dd4ac7c1135dbb4d855 (127.27.153.125:45527)
I20260810 03:50:54.446147 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:54.445570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.0.0.1:48824)
0810 03:50:54.445963 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:54.446035 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":84,"threads_started":1}
I20260810 03:50:54.446161 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:54.445628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.0.0.1:48836)
0810 03:50:54.445949 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:54.446083 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":68,"threads_started":1}
W20260810 03:50:54.446324 7841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
W20260810 03:50:54.446324 8174 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
I20260810 03:50:54.447276 9114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:54.447054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.0.0.1:48852)
0810 03:50:54.447176 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:54.447216 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 03:50:54.447296 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:54.447011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.0.0.1:48850)
0810 03:50:54.447162 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:54.447211 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260810 03:50:54.447413 8174 leader_election.cc:334] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f3d2c2bbe823423ca1284d9fa2a52b2b (127.27.153.126:41933): Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
W20260810 03:50:54.447413 7841 leader_election.cc:334] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f3d2c2bbe823423ca1284d9fa2a52b2b (127.27.153.126:41933): Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
I20260810 03:50:54.450495 9115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:54.445852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40226 (local address 127.27.153.124:39435)
0810 03:50:54.446293 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:54.446297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:54.446499 (+ 202us) server_negotiation.cc:407] Connection header received
0810 03:50:54.446711 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:54.446714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:54.446755 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:54.446810 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:54.447197 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:54.448541 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:54.449570 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:54.449969 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:54.450069 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:54.450121 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:54.450267 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 03:50:54.450371 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":386,"thread_start_us":49,"threads_started":1}
I20260810 03:50:54.450490 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:54.446351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47722 (local address 127.27.153.125:45527)
0810 03:50:54.446658 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:54.446662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:54.446673 (+ 11us) server_negotiation.cc:407] Connection header received
0810 03:50:54.446707 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:54.446710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:54.446750 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:54.447061 (+ 311us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:54.447131 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:54.448353 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:54.449515 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:54.449939 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:54.450049 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:54.450117 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:54.450262 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 03:50:54.450342 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":250,"thread_start_us":69,"threads_started":1}
I20260810 03:50:54.450490 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:54.446004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:39435 (local address 127.0.0.1:40226)
0810 03:50:54.446423 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:54.446436 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:54.446573 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:54.446957 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:54.446959 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:54.446975 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:54.447072 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:54.447078 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:54.448658 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:54.448661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:54.449468 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:54.449481 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:54.450070 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:54.450074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:54.450161 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:54.450179 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 03:50:54.450244 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 03:50:54.450340 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":70,"threads_started":1}
I20260810 03:50:54.450498 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:54.445616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:45527 (local address 127.0.0.1:47722)
0810 03:50:54.446000 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:54.446014 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:54.446125 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:54.446862 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:54.446865 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:54.446878 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:54.446969 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:54.446975 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:54.448477 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:54.448480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:54.449386 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:54.449397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:54.450061 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:54.450065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:54.450152 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:54.450171 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 03:50:54.450235 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 03:50:54.450347 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":3,"thread_start_us":103,"threads_started":1}
I20260810 03:50:54.451079 7875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0fed699bb95348be8b9a62282240a7de" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" is_pre_election: true
I20260810 03:50:54.451079 8221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0fed699bb95348be8b9a62282240a7de" is_pre_election: true
I20260810 03:50:54.451305 8221 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e655481b21e4dd4ac7c1135dbb4d855 in term 1.
I20260810 03:50:54.451305 7875 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fed699bb95348be8b9a62282240a7de in term 1.
I20260810 03:50:54.451712 7844 leader_election.cc:302] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fed699bb95348be8b9a62282240a7de, 5e655481b21e4dd4ac7c1135dbb4d855; no voters: f3d2c2bbe823423ca1284d9fa2a52b2b
I20260810 03:50:54.451736 8179 leader_election.cc:302] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fed699bb95348be8b9a62282240a7de, 5e655481b21e4dd4ac7c1135dbb4d855; no voters: f3d2c2bbe823423ca1284d9fa2a52b2b
I20260810 03:50:54.451961 9111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 03:50:54.452023 9111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 1 FOLLOWER]: Starting leader election (detected failure of leader f3d2c2bbe823423ca1284d9fa2a52b2b)
I20260810 03:50:54.452068 9111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:50:54.452126 9110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 03:50:54.452196 9110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f3d2c2bbe823423ca1284d9fa2a52b2b)
I20260810 03:50:54.452240 9110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:50:54.453231 9110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: false } }
I20260810 03:50:54.453230 9111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: false } }
I20260810 03:50:54.453614 9111 leader_election.cc:288] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [CANDIDATE]: Term 2 election: Requested vote from peers f3d2c2bbe823423ca1284d9fa2a52b2b (127.27.153.126:41933), 5e655481b21e4dd4ac7c1135dbb4d855 (127.27.153.125:45527)
I20260810 03:50:54.453740 9110 leader_election.cc:288] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [CANDIDATE]: Term 2 election: Requested vote from peers f3d2c2bbe823423ca1284d9fa2a52b2b (127.27.153.126:41933), 0fed699bb95348be8b9a62282240a7de (127.27.153.124:39435)
I20260810 03:50:54.454131 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:54.453790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.0.0.1:48876)
0810 03:50:54.453997 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:54.454054 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260810 03:50:54.454130 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:54.453892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.0.0.1:48882)
0810 03:50:54.454033 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:54.454075 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260810 03:50:54.454186 7875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0fed699bb95348be8b9a62282240a7de" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5e655481b21e4dd4ac7c1135dbb4d855"
I20260810 03:50:54.454296 8221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0fed699bb95348be8b9a62282240a7de"
I20260810 03:50:54.454380 7875 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0fed699bb95348be8b9a62282240a7de in current term 2: Already voted for candidate 5e655481b21e4dd4ac7c1135dbb4d855 in this term.
I20260810 03:50:54.454437 8221 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5e655481b21e4dd4ac7c1135dbb4d855 in current term 2: Already voted for candidate 0fed699bb95348be8b9a62282240a7de in this term.
I20260810 03:50:54.454895 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:54.454636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.0.0.1:48888)
0810 03:50:54.454785 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:54.454837 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260810 03:50:54.455000 8174 leader_election.cc:334] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f3d2c2bbe823423ca1284d9fa2a52b2b (127.27.153.126:41933): Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
I20260810 03:50:54.455072 8174 leader_election.cc:302] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0fed699bb95348be8b9a62282240a7de; no voters: 5e655481b21e4dd4ac7c1135dbb4d855, f3d2c2bbe823423ca1284d9fa2a52b2b
I20260810 03:50:54.455211 9111 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 03:50:54.455359 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:54.455133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.0.0.1:48898)
0810 03:50:54.455229 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:54.455295 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 03:50:54.455489 7841 leader_election.cc:334] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f3d2c2bbe823423ca1284d9fa2a52b2b (127.27.153.126:41933): Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
I20260810 03:50:54.455569 7841 leader_election.cc:302] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e655481b21e4dd4ac7c1135dbb4d855; no voters: 0fed699bb95348be8b9a62282240a7de, f3d2c2bbe823423ca1284d9fa2a52b2b
I20260810 03:50:54.455714 9110 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 03:50:54.929352 9122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:54.928892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.65:57211)
0810 03:50:54.929171 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:54.929237 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":112,"threads_started":1}
I20260810 03:50:54.934912 9126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:54.934449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.67:42641)
0810 03:50:54.934729 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:54.934804 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":94,"threads_started":1}
I20260810 03:50:54.934952 9125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:54.934341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.66:53083)
0810 03:50:54.934810 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:54.934868 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":101,"threads_started":1}
I20260810 03:50:55.930832 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:55.930333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.65:46381)
0810 03:50:55.930641 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:55.930719 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":84,"threads_started":1}
I20260810 03:50:55.936028 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:55.935580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.67:49503)
0810 03:50:55.935845 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:55.935921 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":66,"threads_started":1}
I20260810 03:50:55.936038 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:55.935580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.66:35887)
0810 03:50:55.935888 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:55.935953 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":82,"threads_started":1}
I20260810 03:50:56.226980 9130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:50:56.227133 9130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: false } }
I20260810 03:50:56.227626 9130 leader_election.cc:288] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f3d2c2bbe823423ca1284d9fa2a52b2b (127.27.153.126:41933), 5e655481b21e4dd4ac7c1135dbb4d855 (127.27.153.125:45527)
I20260810 03:50:56.228190 7875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0fed699bb95348be8b9a62282240a7de" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" is_pre_election: true
I20260810 03:50:56.228317 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.227834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.0.0.1:49154)
0810 03:50:56.228128 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.228208 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":99,"threads_started":1}
I20260810 03:50:56.228381 7875 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fed699bb95348be8b9a62282240a7de in term 2.
I20260810 03:50:56.228736 8179 leader_election.cc:302] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fed699bb95348be8b9a62282240a7de, 5e655481b21e4dd4ac7c1135dbb4d855; no voters:
I20260810 03:50:56.228922 9130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 03:50:56.228999 9130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:50:56.229084 9130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 2 FOLLOWER]: Advancing to term 3
I20260810 03:50:56.229242 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.228981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.0.0.1:49168)
0810 03:50:56.229109 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.229173 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 03:50:56.229418 8174 leader_election.cc:334] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f3d2c2bbe823423ca1284d9fa2a52b2b (127.27.153.126:41933): Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
I20260810 03:50:56.230279 9130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: false } }
I20260810 03:50:56.230659 9130 leader_election.cc:288] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [CANDIDATE]: Term 3 election: Requested vote from peers f3d2c2bbe823423ca1284d9fa2a52b2b (127.27.153.126:41933), 5e655481b21e4dd4ac7c1135dbb4d855 (127.27.153.125:45527)
I20260810 03:50:56.231024 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.230790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.0.0.1:49182)
0810 03:50:56.230903 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.230951 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 03:50:56.231086 7875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0fed699bb95348be8b9a62282240a7de" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5e655481b21e4dd4ac7c1135dbb4d855"
I20260810 03:50:56.231228 7875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 2 FOLLOWER]: Advancing to term 3
I20260810 03:50:56.231729 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.231515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.0.0.1:49188)
0810 03:50:56.231616 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.231671 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 03:50:56.231837 8174 leader_election.cc:334] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f3d2c2bbe823423ca1284d9fa2a52b2b (127.27.153.126:41933): Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
I20260810 03:50:56.232424 7875 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fed699bb95348be8b9a62282240a7de in term 3.
I20260810 03:50:56.232743 8179 leader_election.cc:302] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0fed699bb95348be8b9a62282240a7de, 5e655481b21e4dd4ac7c1135dbb4d855; no voters: f3d2c2bbe823423ca1284d9fa2a52b2b
I20260810 03:50:56.232908 9130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 3 FOLLOWER]: Leader election won for term 3
I20260810 03:50:56.233098 9130 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [term 3 LEADER]: Becoming Leader. State: Replica: 0fed699bb95348be8b9a62282240a7de, State: Running, Role: LEADER
I20260810 03:50:56.233351 9130 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: false } }
I20260810 03:50:56.234586 9133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fed699bb95348be8b9a62282240a7de. Latest consensus state: current_term: 3 leader_uuid: "0fed699bb95348be8b9a62282240a7de" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: false } } }
I20260810 03:50:56.234726 9133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:56.235082 9138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:50:56.237795 9138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:50:56.238206 9138 catalog_manager.cc:1259] Loaded cluster ID: f35189ad36ec45d1a449e600cbd8824f
I20260810 03:50:56.238250 9138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:50:56.238806 9138 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:50:56.239179 9138 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de: Loaded TSK: 0
I20260810 03:50:56.239969 9138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:50:56.285424 8191 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:50:56.288574 8191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:50:56.291741 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.291437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.0.0.1:49204)
0810 03:50:56.291582 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.291650 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 03:50:56.291909 7875 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [term 3 FOLLOWER]: Refusing update from remote peer 0fed699bb95348be8b9a62282240a7de: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260810 03:50:56.292460 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.292209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.0.0.1:49214)
0810 03:50:56.292333 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.292398 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 03:50:56.292439 9133 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260810 03:50:56.292636 8174 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de -> Peer f3d2c2bbe823423ca1284d9fa2a52b2b (127.27.153.126:41933): Couldn't send request to peer f3d2c2bbe823423ca1284d9fa2a52b2b. Status: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 03:50:56.294018 9142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fed699bb95348be8b9a62282240a7de. Latest consensus state: current_term: 3 leader_uuid: "0fed699bb95348be8b9a62282240a7de" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: false } } }
I20260810 03:50:56.294166 9142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:50:56.295274 9133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "0fed699bb95348be8b9a62282240a7de" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: false } } }
I20260810 03:50:56.295384 9133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:56.295647 9142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "0fed699bb95348be8b9a62282240a7de" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f3d2c2bbe823423ca1284d9fa2a52b2b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41933 } } peers { permanent_uuid: "5e655481b21e4dd4ac7c1135dbb4d855" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45527 } attrs { promote: false } } peers { permanent_uuid: "0fed699bb95348be8b9a62282240a7de" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 39435 } attrs { promote: false } } }
I20260810 03:50:56.295774 9142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e655481b21e4dd4ac7c1135dbb4d855 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:50:56.305830 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.301459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:41001 (local address 127.0.0.1:33490)
0810 03:50:56.301617 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.301632 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:56.301729 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:56.302236 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:56.302239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:56.302257 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:56.302374 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:56.302380 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:56.303374 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:56.303377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:56.304521 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:56.304531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:56.305329 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:56.305332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:56.305431 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:56.305538 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 03:50:56.305603 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 03:50:56.305682 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":42}
I20260810 03:50:56.306211 9157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.301877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:35535 (local address 127.0.0.1:40230)
0810 03:50:56.302116 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.302129 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:56.302223 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:56.302886 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:56.302889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:56.302905 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:56.303005 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:56.303011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:56.303890 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:56.303894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:56.304869 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:56.304879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:56.305835 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:56.305838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:56.305927 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:56.305948 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 03:50:56.306011 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 03:50:56.306075 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":1,"thread_start_us":110,"threads_started":1}
I20260810 03:50:56.306321 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.301571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33490 (local address 127.27.153.66:41001)
0810 03:50:56.301914 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:56.301918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:56.301931 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:50:56.301982 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:56.301985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:56.302032 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:56.302116 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:56.302520 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:56.303242 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:56.304633 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:56.305087 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:56.305203 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:56.305272 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:56.306107 (+ 835us) server_negotiation.cc:299] Negotiation successful
0810 03:50:56.306205 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":153,"threads_started":1}
I20260810 03:50:56.306581 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.301894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40230 (local address 127.27.153.65:35535)
0810 03:50:56.302433 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:56.302437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:56.302450 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:50:56.302724 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:56.302728 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:56.302779 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:56.302863 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:56.303111 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:56.303787 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:56.305010 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:56.305617 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:56.305705 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:56.305779 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:56.306371 (+ 592us) server_negotiation.cc:299] Negotiation successful
0810 03:50:56.306454 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":88,"threads_started":1}
I20260810 03:50:56.306744 9164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.302199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:45049 (local address 127.0.0.1:47770)
0810 03:50:56.302433 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.302446 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:56.302531 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:56.303227 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:56.303230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:56.303248 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:56.303350 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:56.303355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:56.304371 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:56.304374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:56.305528 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:56.305539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:56.306401 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:56.306405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:56.306485 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:56.306507 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 03:50:56.306552 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 03:50:56.306616 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":36,"thread_start_us":97,"threads_started":1}
I20260810 03:50:56.306891 8627 tablet_service.cc:1467] Processing CreateTablet for tablet a4dee453c1e44578a5b6a69148417b15 (DEFAULT_TABLE table=test-table [id=c5029409d5734d82bd136b4feadd3bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:56.307118 8389 tablet_service.cc:1467] Processing CreateTablet for tablet a4dee453c1e44578a5b6a69148417b15 (DEFAULT_TABLE table=test-table [id=c5029409d5734d82bd136b4feadd3bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:56.307228 8627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4dee453c1e44578a5b6a69148417b15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:56.307242 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.302638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47770 (local address 127.27.153.67:45049)
0810 03:50:56.303022 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:56.303027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:56.303040 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:50:56.303087 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:56.303089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:56.303130 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:56.303210 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:56.303447 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:56.304252 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:56.305670 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:56.306271 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:56.306964 (+ 693us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:56.307020 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:56.307057 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 03:50:56.307133 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":95,"threads_started":1}
I20260810 03:50:56.307387 8389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4dee453c1e44578a5b6a69148417b15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:56.307725 8862 tablet_service.cc:1467] Processing CreateTablet for tablet a4dee453c1e44578a5b6a69148417b15 (DEFAULT_TABLE table=test-table [id=c5029409d5734d82bd136b4feadd3bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:56.307974 8862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4dee453c1e44578a5b6a69148417b15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:56.310741 9167 tablet_bootstrap.cc:492] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f: Bootstrap starting.
I20260810 03:50:56.311642 9169 tablet_bootstrap.cc:492] T a4dee453c1e44578a5b6a69148417b15 P 16dc897a7f744e78a9b0e59c3abd6d11: Bootstrap starting.
I20260810 03:50:56.311738 9167 tablet_bootstrap.cc:654] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:56.312500 9169 tablet_bootstrap.cc:654] T a4dee453c1e44578a5b6a69148417b15 P 16dc897a7f744e78a9b0e59c3abd6d11: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:56.312912 9167 tablet_bootstrap.cc:492] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f: No bootstrap required, opened a new log
I20260810 03:50:56.313011 9167 ts_tablet_manager.cc:1397] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 03:50:56.313095 9168 tablet_bootstrap.cc:492] T a4dee453c1e44578a5b6a69148417b15 P 6720a63653bd424bba05965f0b0844dc: Bootstrap starting.
I20260810 03:50:56.313522 9167 raft_consensus.cc:348] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a083bc0c0ba48e79563564743d2fb3f" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 41001 } } peers { permanent_uuid: "6720a63653bd424bba05965f0b0844dc" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 35535 } } peers { permanent_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 45049 } }
I20260810 03:50:56.313630 9167 raft_consensus.cc:374] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:56.313670 9167 raft_consensus.cc:729] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a083bc0c0ba48e79563564743d2fb3f, State: Initialized, Role: FOLLOWER
I20260810 03:50:56.313843 9169 tablet_bootstrap.cc:492] T a4dee453c1e44578a5b6a69148417b15 P 16dc897a7f744e78a9b0e59c3abd6d11: No bootstrap required, opened a new log
I20260810 03:50:56.313840 9167 consensus_queue.cc:260] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a083bc0c0ba48e79563564743d2fb3f" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 41001 } } peers { permanent_uuid: "6720a63653bd424bba05965f0b0844dc" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 35535 } } peers { permanent_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 45049 } }
I20260810 03:50:56.313941 9169 ts_tablet_manager.cc:1397] T a4dee453c1e44578a5b6a69148417b15 P 16dc897a7f744e78a9b0e59c3abd6d11: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 03:50:56.313993 9168 tablet_bootstrap.cc:654] T a4dee453c1e44578a5b6a69148417b15 P 6720a63653bd424bba05965f0b0844dc: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:56.314179 9167 ts_tablet_manager.cc:1428] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:50:56.314445 9169 raft_consensus.cc:348] T a4dee453c1e44578a5b6a69148417b15 P 16dc897a7f744e78a9b0e59c3abd6d11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a083bc0c0ba48e79563564743d2fb3f" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 41001 } } peers { permanent_uuid: "6720a63653bd424bba05965f0b0844dc" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 35535 } } peers { permanent_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 45049 } }
I20260810 03:50:56.314594 9169 raft_consensus.cc:374] T a4dee453c1e44578a5b6a69148417b15 P 16dc897a7f744e78a9b0e59c3abd6d11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:56.314649 9169 raft_consensus.cc:729] T a4dee453c1e44578a5b6a69148417b15 P 16dc897a7f744e78a9b0e59c3abd6d11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16dc897a7f744e78a9b0e59c3abd6d11, State: Initialized, Role: FOLLOWER
I20260810 03:50:56.314828 9169 consensus_queue.cc:260] T a4dee453c1e44578a5b6a69148417b15 P 16dc897a7f744e78a9b0e59c3abd6d11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a083bc0c0ba48e79563564743d2fb3f" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 41001 } } peers { permanent_uuid: "6720a63653bd424bba05965f0b0844dc" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 35535 } } peers { permanent_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 45049 } }
I20260810 03:50:56.315233 9169 ts_tablet_manager.cc:1428] T a4dee453c1e44578a5b6a69148417b15 P 16dc897a7f744e78a9b0e59c3abd6d11: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:50:56.316283 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.316001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.67:41101)
0810 03:50:56.316137 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.316201 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 03:50:56.316939 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.315107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.66:38487)
0810 03:50:56.316824 (+ 1717us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.316879 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1637}
I20260810 03:50:56.317448 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.317218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.67:59401)
0810 03:50:56.317333 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.317384 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 03:50:56.318276 8742 heartbeater.cc:503] Master 127.27.153.124:39435 was elected leader, sending a full tablet report...
I20260810 03:50:56.318745 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.318138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.66:52903)
0810 03:50:56.318608 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.318675 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402}
I20260810 03:50:56.319572 9168 tablet_bootstrap.cc:492] T a4dee453c1e44578a5b6a69148417b15 P 6720a63653bd424bba05965f0b0844dc: No bootstrap required, opened a new log
I20260810 03:50:56.319741 9168 ts_tablet_manager.cc:1397] T a4dee453c1e44578a5b6a69148417b15 P 6720a63653bd424bba05965f0b0844dc: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260810 03:50:56.320197 9168 raft_consensus.cc:348] T a4dee453c1e44578a5b6a69148417b15 P 6720a63653bd424bba05965f0b0844dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a083bc0c0ba48e79563564743d2fb3f" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 41001 } } peers { permanent_uuid: "6720a63653bd424bba05965f0b0844dc" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 35535 } } peers { permanent_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 45049 } }
I20260810 03:50:56.320304 9168 raft_consensus.cc:374] T a4dee453c1e44578a5b6a69148417b15 P 6720a63653bd424bba05965f0b0844dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:56.320334 9168 raft_consensus.cc:729] T a4dee453c1e44578a5b6a69148417b15 P 6720a63653bd424bba05965f0b0844dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6720a63653bd424bba05965f0b0844dc, State: Initialized, Role: FOLLOWER
I20260810 03:50:56.320468 9168 consensus_queue.cc:260] T a4dee453c1e44578a5b6a69148417b15 P 6720a63653bd424bba05965f0b0844dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a083bc0c0ba48e79563564743d2fb3f" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 41001 } } peers { permanent_uuid: "6720a63653bd424bba05965f0b0844dc" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 35535 } } peers { permanent_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 45049 } }
I20260810 03:50:56.320811 9168 ts_tablet_manager.cc:1428] T a4dee453c1e44578a5b6a69148417b15 P 6720a63653bd424bba05965f0b0844dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:50:56.321375 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.321121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.65:37787)
0810 03:50:56.321243 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.321304 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 03:50:56.322232 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.321811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.65:49149)
0810 03:50:56.322103 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.322155 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221}
I20260810 03:50:56.322576 8507 heartbeater.cc:503] Master 127.27.153.124:39435 was elected leader, sending a full tablet report...
I20260810 03:50:56.449422 9180 raft_consensus.cc:482] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:50:56.449594 9180 raft_consensus.cc:504] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a083bc0c0ba48e79563564743d2fb3f" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 41001 } } peers { permanent_uuid: "6720a63653bd424bba05965f0b0844dc" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 35535 } } peers { permanent_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 45049 } }
I20260810 03:50:56.450747 9180 leader_election.cc:288] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6720a63653bd424bba05965f0b0844dc (127.27.153.65:35535), 16dc897a7f744e78a9b0e59c3abd6d11 (127.27.153.67:45049)
I20260810 03:50:56.454860 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.451175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:44589 (local address 127.27.153.65:35535)
0810 03:50:56.451321 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:56.451326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:56.451342 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:50:56.451617 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:56.451621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:56.451672 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:56.451743 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:56.452041 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:56.452560 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:56.453681 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:56.454258 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:56.454380 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:56.454436 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:56.454645 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 03:50:56.454731 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 03:50:56.454851 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.451050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:35535 (local address 127.27.153.66:44589)
0810 03:50:56.451186 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.451201 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:56.451324 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:56.451775 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:56.451778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:56.451799 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:56.451931 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:56.451936 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:56.452665 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:56.452668 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:56.453513 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:56.453524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:56.454392 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:56.454396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:56.454503 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:56.454531 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 03:50:56.454613 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 03:50:56.454697 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":1,"spinlock_wait_cycles":14208}
I20260810 03:50:56.455212 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.451494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:60655 (local address 127.27.153.67:45049)
0810 03:50:56.451618 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:56.451622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:56.451813 (+ 191us) server_negotiation.cc:407] Connection header received
0810 03:50:56.451959 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:56.451962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:56.452006 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:56.452066 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:56.452326 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:56.452878 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:56.453983 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:56.454545 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:56.454650 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:56.454694 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:56.455016 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 03:50:56.455103 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 03:50:56.455617 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.451346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:45049 (local address 127.27.153.66:60655)
0810 03:50:56.451711 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.451723 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:56.451856 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:56.452088 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:56.452091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:56.452106 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:56.452218 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:56.452223 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:56.452986 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:56.452989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:56.453863 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:56.453874 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:56.454764 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:56.454768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:56.454864 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:56.454886 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 03:50:56.455359 (+ 473us) client_negotiation.cc:240] Negotiation successful
0810 03:50:56.455439 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":22,"spinlock_wait_cycles":1920,"thread_start_us":159,"threads_started":1}
I20260810 03:50:56.455648 8428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4dee453c1e44578a5b6a69148417b15" candidate_uuid: "7a083bc0c0ba48e79563564743d2fb3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6720a63653bd424bba05965f0b0844dc" is_pre_election: true
I20260810 03:50:56.455812 8428 raft_consensus.cc:2454] T a4dee453c1e44578a5b6a69148417b15 P 6720a63653bd424bba05965f0b0844dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a083bc0c0ba48e79563564743d2fb3f in term 0.
I20260810 03:50:56.456171 8550 leader_election.cc:302] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6720a63653bd424bba05965f0b0844dc, 7a083bc0c0ba48e79563564743d2fb3f; no voters:
I20260810 03:50:56.456259 8901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4dee453c1e44578a5b6a69148417b15" candidate_uuid: "7a083bc0c0ba48e79563564743d2fb3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" is_pre_election: true
I20260810 03:50:56.456427 9180 raft_consensus.cc:2781] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:50:56.456492 9180 raft_consensus.cc:482] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:50:56.456537 9180 raft_consensus.cc:3037] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:56.456513 8901 raft_consensus.cc:2454] T a4dee453c1e44578a5b6a69148417b15 P 16dc897a7f744e78a9b0e59c3abd6d11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a083bc0c0ba48e79563564743d2fb3f in term 0.
I20260810 03:50:56.459322 9180 raft_consensus.cc:504] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a083bc0c0ba48e79563564743d2fb3f" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 41001 } } peers { permanent_uuid: "6720a63653bd424bba05965f0b0844dc" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 35535 } } peers { permanent_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 45049 } }
I20260810 03:50:56.459748 9180 leader_election.cc:288] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f [CANDIDATE]: Term 1 election: Requested vote from peers 6720a63653bd424bba05965f0b0844dc (127.27.153.65:35535), 16dc897a7f744e78a9b0e59c3abd6d11 (127.27.153.67:45049)
I20260810 03:50:56.460126 8428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4dee453c1e44578a5b6a69148417b15" candidate_uuid: "7a083bc0c0ba48e79563564743d2fb3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6720a63653bd424bba05965f0b0844dc"
I20260810 03:50:56.460224 8901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4dee453c1e44578a5b6a69148417b15" candidate_uuid: "7a083bc0c0ba48e79563564743d2fb3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16dc897a7f744e78a9b0e59c3abd6d11"
I20260810 03:50:56.460278 8428 raft_consensus.cc:3037] T a4dee453c1e44578a5b6a69148417b15 P 6720a63653bd424bba05965f0b0844dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:56.460350 8901 raft_consensus.cc:3037] T a4dee453c1e44578a5b6a69148417b15 P 16dc897a7f744e78a9b0e59c3abd6d11 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:56.461241 8428 raft_consensus.cc:2454] T a4dee453c1e44578a5b6a69148417b15 P 6720a63653bd424bba05965f0b0844dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a083bc0c0ba48e79563564743d2fb3f in term 1.
I20260810 03:50:56.461283 8901 raft_consensus.cc:2454] T a4dee453c1e44578a5b6a69148417b15 P 16dc897a7f744e78a9b0e59c3abd6d11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a083bc0c0ba48e79563564743d2fb3f in term 1.
I20260810 03:50:56.461531 8550 leader_election.cc:302] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6720a63653bd424bba05965f0b0844dc, 7a083bc0c0ba48e79563564743d2fb3f; no voters:
I20260810 03:50:56.461745 9180 raft_consensus.cc:2781] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:56.461973 9180 raft_consensus.cc:686] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f [term 1 LEADER]: Becoming Leader. State: Replica: 7a083bc0c0ba48e79563564743d2fb3f, State: Running, Role: LEADER
I20260810 03:50:56.462405 9180 consensus_queue.cc:237] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a083bc0c0ba48e79563564743d2fb3f" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 41001 } } peers { permanent_uuid: "6720a63653bd424bba05965f0b0844dc" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 35535 } } peers { permanent_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 45049 } }
I20260810 03:50:56.462672 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.462425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.66:46659)
0810 03:50:56.462540 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.462598 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 03:50:56.465341 8204 catalog_manager.cc:5495] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a083bc0c0ba48e79563564743d2fb3f (127.27.153.66). New cstate: current_term: 1 leader_uuid: "7a083bc0c0ba48e79563564743d2fb3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a083bc0c0ba48e79563564743d2fb3f" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 41001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6720a63653bd424bba05965f0b0844dc" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 35535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 45049 } health_report { overall_health: UNKNOWN } } }
I20260810 03:50:56.500383 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.496813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33500 (local address 127.27.153.66:41001)
0810 03:50:56.496953 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:56.496958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:56.496975 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:50:56.497065 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:56.497068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:56.497138 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:56.497238 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:50:56.497505 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:56.498020 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:56.498874 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:56.499088 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:56.499190 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:56.499504 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:50:56.499976 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:50:56.500087 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:56.500173 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 03:50:56.500239 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:50:56.503268 8428 raft_consensus.cc:1261] T a4dee453c1e44578a5b6a69148417b15 P 6720a63653bd424bba05965f0b0844dc [term 1 FOLLOWER]: Refusing update from remote peer 7a083bc0c0ba48e79563564743d2fb3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:56.503590 8901 raft_consensus.cc:1261] T a4dee453c1e44578a5b6a69148417b15 P 16dc897a7f744e78a9b0e59c3abd6d11 [term 1 FOLLOWER]: Refusing update from remote peer 7a083bc0c0ba48e79563564743d2fb3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:56.503899 9189 consensus_queue.cc:1035] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6720a63653bd424bba05965f0b0844dc" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 35535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:56.504122 9180 consensus_queue.cc:1035] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "16dc897a7f744e78a9b0e59c3abd6d11" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 45049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:56.506103 8982 heartbeater.cc:503] Master 127.27.153.124:39435 was elected leader, sending a full tablet report...
I20260810 03:50:56.506482 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.506162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.65:33155)
0810 03:50:56.506329 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.506398 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260810 03:50:56.508385 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.508109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.67:34883)
0810 03:50:56.508263 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.508315 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260810 03:50:56.510488 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.509542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.66:46789)
0810 03:50:56.510346 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.510408 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":719}
I20260810 03:50:56.511225 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.510970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.66:39887)
0810 03:50:56.511101 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.511149 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 03:50:56.512890 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.512634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.65:43839)
0810 03:50:56.512781 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.512829 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260810 03:50:56.513309 9129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.511878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:41933 (local address 127.27.153.67:43751)
0810 03:50:56.513201 (+ 1323us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.513251 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1252}
I20260810 03:50:56.521948 8191 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:56.522137 8191 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:56.526584 8191 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0fed699bb95348be8b9a62282240a7de: Sending DeleteTablet for 3 replicas of tablet a4dee453c1e44578a5b6a69148417b15
I20260810 03:50:56.527375 8627 tablet_service.cc:1514] Processing DeleteTablet for tablet a4dee453c1e44578a5b6a69148417b15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:56 UTC) from {username='oryx', principal='oryx/127.27.153.124@KRBTEST.COM'} at 127.0.0.1:33490
I20260810 03:50:56.527733 8389 tablet_service.cc:1514] Processing DeleteTablet for tablet a4dee453c1e44578a5b6a69148417b15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:56 UTC) from {username='oryx', principal='oryx/127.27.153.124@KRBTEST.COM'} at 127.0.0.1:40230
I20260810 03:50:56.528146 9217 tablet_replica.cc:330] T a4dee453c1e44578a5b6a69148417b15 P 6720a63653bd424bba05965f0b0844dc: stopping tablet replica
I20260810 03:50:56.528156 8862 tablet_service.cc:1514] Processing DeleteTablet for tablet a4dee453c1e44578a5b6a69148417b15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:56 UTC) from {username='oryx', principal='oryx/127.27.153.124@KRBTEST.COM'} at 127.0.0.1:47770
I20260810 03:50:56.528403 9217 raft_consensus.cc:2229] T a4dee453c1e44578a5b6a69148417b15 P 6720a63653bd424bba05965f0b0844dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:56.528581 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 8129
I20260810 03:50:56.530112 9215 tablet_replica.cc:330] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f: stopping tablet replica
I20260810 03:50:56.530280 9215 raft_consensus.cc:2229] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:50:56.530726 9215 raft_consensus.cc:2258] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:56.533104 9215 ts_tablet_manager.cc:1905] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:56.537562 9215 ts_tablet_manager.cc:1918] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:50:56.537684 9215 log.cc:1196] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/a4dee453c1e44578a5b6a69148417b15
I20260810 03:50:56.538033 9215 ts_tablet_manager.cc:1939] T a4dee453c1e44578a5b6a69148417b15 P 7a083bc0c0ba48e79563564743d2fb3f: Deleting consensus metadata
I20260810 03:50:56.545156 8175 catalog_manager.cc:4841] TS 7a083bc0c0ba48e79563564743d2fb3f (127.27.153.66:41001): tablet a4dee453c1e44578a5b6a69148417b15 (table test-table [id=c5029409d5734d82bd136b4feadd3bcf]) successfully deleted
I20260810 03:50:56.547330 9218 tablet_replica.cc:330] T a4dee453c1e44578a5b6a69148417b15 P 16dc897a7f744e78a9b0e59c3abd6d11: stopping tablet replica
I20260810 03:50:56.547590 9218 raft_consensus.cc:2229] T a4dee453c1e44578a5b6a69148417b15 P 16dc897a7f744e78a9b0e59c3abd6d11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:56.547832 9218 raft_consensus.cc:2258] T a4dee453c1e44578a5b6a69148417b15 P 16dc897a7f744e78a9b0e59c3abd6d11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:56.548820 9218 ts_tablet_manager.cc:1905] T a4dee453c1e44578a5b6a69148417b15 P 16dc897a7f744e78a9b0e59c3abd6d11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:56.551884 9218 ts_tablet_manager.cc:1918] T a4dee453c1e44578a5b6a69148417b15 P 16dc897a7f744e78a9b0e59c3abd6d11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:50:56.552052 9218 log.cc:1196] T a4dee453c1e44578a5b6a69148417b15 P 16dc897a7f744e78a9b0e59c3abd6d11: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/a4dee453c1e44578a5b6a69148417b15
I20260810 03:50:56.552435 9218 ts_tablet_manager.cc:1939] T a4dee453c1e44578a5b6a69148417b15 P 16dc897a7f744e78a9b0e59c3abd6d11: Deleting consensus metadata
I20260810 03:50:56.552943 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 8518
I20260810 03:50:56.553256 9164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:56.552881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:35535 (local address 127.0.0.1:40238)
0810 03:50:56.553101 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:56.553165 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260810 03:50:56.567443 8174 catalog_manager.cc:4584] TS 6720a63653bd424bba05965f0b0844dc (127.27.153.65:35535): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4dee453c1e44578a5b6a69148417b15: Network error: Client connection negotiation failed: client connection to 127.27.153.65:35535: connect: Connection refused (error 111)
I20260810 03:50:56.567816 8174 catalog_manager.cc:4841] TS 16dc897a7f744e78a9b0e59c3abd6d11 (127.27.153.67:45049): tablet a4dee453c1e44578a5b6a69148417b15 (table test-table [id=c5029409d5734d82bd136b4feadd3bcf]) successfully deleted
I20260810 03:50:56.573601 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 8750
I20260810 03:50:56.591868 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 7619
I20260810 03:50:56.610419 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 7804
2026-08-10T03:50:56Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7898 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:50:56 dist-test-slave-pww0 krb5kdc[9222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:50:56 dist-test-slave-pww0 krb5kdc[9222](info): set up 2 sockets
Aug 10 03:50:56 dist-test-slave-pww0 krb5kdc[9222](info): commencing operation
krb5kdc: starting...
W20260810 03:50:57.831207 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:50:57 dist-test-slave-pww0 krb5kdc[9222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:50:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:50:57Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.126.keytab.
Entry for principal oryx/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.126.keytab.
I20260810 03:50:57.965001 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:37893
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.126.keytab
--principal=oryx/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:34857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:37893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:58.084408 9241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:58.084707 9241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:58.084748 9241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:58.088855 9241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:50:58.088930 9241 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 03:50:58.088948 9241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:58.088964 9241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:58.088979 9241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:50:58.088992 9241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:50:58.094774 9241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:34857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:37893
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.126.keytab
--principal=oryx/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:37893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:58.095958 9241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:58.097036 9241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:58.106122 9259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:58.106179 9254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:58.106133 9256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:58.106359 9241 server_base.cc:1034] running on GCE node
I20260810 03:50:58.107041 9241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:58.108302 9241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:58.109491 9241 hybrid_clock.cc:648] HybridClock initialized: now 1786333858109473 us; error 30 us; skew 500 ppm
Aug 10 03:50:58 dist-test-slave-pww0 krb5kdc[9222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333858, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:58.112494 9241 init.cc:377] Logged in from keytab as oryx/127.27.153.126@KRBTEST.COM (short username oryx)
I20260810 03:50:58.113524 9241 webserver.cc:460] Webserver started at http://127.27.153.126:35359/ using document root <none> and password file <none>
I20260810 03:50:58.114092 9241 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:58.114166 9241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:58.114393 9241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:58.116647 9241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "648f6a718a104afa8530c6ffc0acbc94"
format_stamp: "Formatted at 2026-08-10 03:50:58 on dist-test-slave-pww0"
server_key: "1d980dee3418d49ce1cd0f85b9fb5f52"
server_key_iv: "118430f118d4f40ddcb0431d3cc6919f"
server_key_version: "encryptionkey@0"
I20260810 03:50:58.117228 9241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "648f6a718a104afa8530c6ffc0acbc94"
format_stamp: "Formatted at 2026-08-10 03:50:58 on dist-test-slave-pww0"
server_key: "1d980dee3418d49ce1cd0f85b9fb5f52"
server_key_iv: "118430f118d4f40ddcb0431d3cc6919f"
server_key_version: "encryptionkey@0"
I20260810 03:50:58.121333 9241 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 03:50:58.124039 9273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:58.125538 9241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:58.125682 9241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "648f6a718a104afa8530c6ffc0acbc94"
format_stamp: "Formatted at 2026-08-10 03:50:58 on dist-test-slave-pww0"
server_key: "1d980dee3418d49ce1cd0f85b9fb5f52"
server_key_iv: "118430f118d4f40ddcb0431d3cc6919f"
server_key_version: "encryptionkey@0"
I20260810 03:50:58.125792 9241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:58.144202 9241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:58.147765 9241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:58.147982 9241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:58.156731 9241 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:37893
I20260810 03:50:58.156810 9383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:37893 every 8 connection(s)
I20260810 03:50:58.157790 9241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:50:58.160928 9384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:58.162873 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 9241
I20260810 03:50:58.162986 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:50:58.163264 28261 external_mini_cluster.cc:1428] Setting key 37b227c41e32feb6cbe725af93d17578
I20260810 03:50:58.168296 9384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94: Bootstrap starting.
I20260810 03:50:58.171222 9384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:58.172083 9384 log.cc:824] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:50:58 dist-test-slave-pww0 krb5kdc[9222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM
I20260810 03:50:58.174451 9384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94: No bootstrap required, opened a new log
I20260810 03:50:58.178365 9384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "648f6a718a104afa8530c6ffc0acbc94" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37893 } }
I20260810 03:50:58.178655 9384 raft_consensus.cc:374] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:58.178728 9384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 648f6a718a104afa8530c6ffc0acbc94, State: Initialized, Role: FOLLOWER
I20260810 03:50:58.179005 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:58.164870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39388 (local address 127.27.153.126:37893)
0810 03:50:58.165344 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:58.165365 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:58.165412 (+ 47us) server_negotiation.cc:407] Connection header received
0810 03:50:58.166232 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:58.166260 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:58.166740 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:58.167151 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:58.167983 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.169454 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.170395 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.171104 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.171270 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:58.173855 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:58.173861 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:58.173877 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:58.173907 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:58.175905 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:58.176381 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:58.176387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:58.176392 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:58.176527 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:58.176814 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:58.176817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:58.176818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:58.177262 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:58.177485 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:58.177780 (+ 295us) server_negotiation.cc:299] Negotiation successful
0810 03:50:58.178104 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":133,"threads_started":1}
I20260810 03:50:58.179253 9384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "648f6a718a104afa8530c6ffc0acbc94" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37893 } }
I20260810 03:50:58.179435 9384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:50:58.179543 9384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:50:58.179643 9384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:58.180825 9384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "648f6a718a104afa8530c6ffc0acbc94" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37893 } }
I20260810 03:50:58.181190 9384 leader_election.cc:302] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 648f6a718a104afa8530c6ffc0acbc94; no voters:
I20260810 03:50:58.181568 9384 leader_election.cc:288] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:50:58.181677 9390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:58.181913 9390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94 [term 1 LEADER]: Becoming Leader. State: Replica: 648f6a718a104afa8530c6ffc0acbc94, State: Running, Role: LEADER
I20260810 03:50:58.182221 9390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "648f6a718a104afa8530c6ffc0acbc94" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37893 } }
I20260810 03:50:58.182865 9384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:50:58.183933 9393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "648f6a718a104afa8530c6ffc0acbc94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "648f6a718a104afa8530c6ffc0acbc94" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37893 } } }
I20260810 03:50:58.184087 9393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:58.184387 9395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 648f6a718a104afa8530c6ffc0acbc94. Latest consensus state: current_term: 1 leader_uuid: "648f6a718a104afa8530c6ffc0acbc94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "648f6a718a104afa8530c6ffc0acbc94" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37893 } } }
I20260810 03:50:58.184485 9395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94 [sys.catalog]: This master's current role is: LEADER
I20260810 03:50:58.184595 9408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:50:58.188030 9408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:50:58.194087 9408 catalog_manager.cc:1347] Generated new cluster ID: ace782ad667a48a69be869bd8809d4bd
I20260810 03:50:58.194200 9408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:50:58.211910 9408 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:50:58.213337 9408 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:50:58.225101 9408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94: Generated new TSK 0
I20260810 03:50:58.225994 9408 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.153.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.153.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.153.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.65.keytab.
Entry for principal oryx/127.27.153.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.65.keytab.
I20260810 03:50:58.267239 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:37893
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.65.keytab
--principal=oryx/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:34857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:58.388605 9430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:58.388918 9430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:58.388964 9430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:58.393113 9430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:58.393188 9430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:58.393234 9430 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:50:58.393268 9430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:50:58.398727 9430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:34857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.65.keytab
--principal=oryx/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.153.126:37893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:58.400040 9430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:58.401228 9430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:58.411296 9447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:58.411309 9455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:58.411330 9443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:58.411592 9430 server_base.cc:1034] running on GCE node
I20260810 03:50:58.412133 9430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:58.412807 9430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:58.413995 9430 hybrid_clock.cc:648] HybridClock initialized: now 1786333858413971 us; error 38 us; skew 500 ppm
Aug 10 03:50:58 dist-test-slave-pww0 krb5kdc[9222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333858, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:58.417068 9430 init.cc:377] Logged in from keytab as oryx/127.27.153.65@KRBTEST.COM (short username oryx)
I20260810 03:50:58.417994 9430 webserver.cc:460] Webserver started at http://127.27.153.65:42231/ using document root <none> and password file <none>
I20260810 03:50:58.418545 9430 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:58.418614 9430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:58.418788 9430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:58.420957 9430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "9950a0f607014fd18fa239f45369f36d"
format_stamp: "Formatted at 2026-08-10 03:50:58 on dist-test-slave-pww0"
server_key: "dea386eb301b23e8426f6a3a4bb0c199"
server_key_iv: "6a21a3fe483e0ffa20da40468b653334"
server_key_version: "encryptionkey@0"
I20260810 03:50:58.421496 9430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "9950a0f607014fd18fa239f45369f36d"
format_stamp: "Formatted at 2026-08-10 03:50:58 on dist-test-slave-pww0"
server_key: "dea386eb301b23e8426f6a3a4bb0c199"
server_key_iv: "6a21a3fe483e0ffa20da40468b653334"
server_key_version: "encryptionkey@0"
I20260810 03:50:58.425796 9430 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:50:58.428625 9468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:58.430073 9430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:50:58.430190 9430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "9950a0f607014fd18fa239f45369f36d"
format_stamp: "Formatted at 2026-08-10 03:50:58 on dist-test-slave-pww0"
server_key: "dea386eb301b23e8426f6a3a4bb0c199"
server_key_iv: "6a21a3fe483e0ffa20da40468b653334"
server_key_version: "encryptionkey@0"
I20260810 03:50:58.430338 9430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:58.442668 9430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:58.446066 9430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:58.446247 9430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:58.447770 9430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:58.447826 9430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:58.447871 9430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:58.447899 9430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:50:58 dist-test-slave-pww0 krb5kdc[9222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333858, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM
I20260810 03:50:58.458189 9430 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:45569
I20260810 03:50:58.458240 9665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:45569 every 8 connection(s)
I20260810 03:50:58.459120 9430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:50:58.462109 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:58.448663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:35405 (local address 127.27.153.126:37893)
0810 03:50:58.448825 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:58.448830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:58.449730 (+ 900us) server_negotiation.cc:407] Connection header received
0810 03:50:58.450888 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:58.450891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:58.450955 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:58.451042 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:58.452900 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.453762 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.455086 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.455384 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.455510 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:58.458323 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:58.458328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:58.458331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:58.458367 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:58.459893 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:58.460637 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:58.460640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:58.460642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:58.460732 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:58.461017 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:58.461019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:58.461020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:58.461259 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:58.461348 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:58.461859 (+ 511us) server_negotiation.cc:299] Negotiation successful
0810 03:50:58.461981 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 03:50:58.462886 9483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:58.449062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:37893 (local address 127.27.153.65:35405)
0810 03:50:58.449542 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:58.449597 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:58.450648 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:58.451309 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:58.451321 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:58.451896 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:58.452664 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:58.452688 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.453941 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.453945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:58.454926 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:58.454940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.455570 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.455575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:58.455775 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:58.456420 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:58.456442 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:58.458098 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:58.460205 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:58.460212 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:58.460226 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:58.460490 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:58.460810 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:58.460812 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:58.460814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:58.460946 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:58.461347 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:58.461353 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:58.461596 (+ 243us) client_negotiation.cc:769] Sending connection context
0810 03:50:58.461776 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 03:50:58.462046 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":115,"threads_started":1}
I20260810 03:50:58.464249 9666 heartbeater.cc:347] Connected to a master server at 127.27.153.126:37893
I20260810 03:50:58.464408 9666 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:58.464941 9666 heartbeater.cc:511] Master 127.27.153.126:37893 requested a full tablet report, sending...
I20260810 03:50:58.465685 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 9430
I20260810 03:50:58.465816 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:50:58.466137 28261 external_mini_cluster.cc:1428] Setting key f489acc11a3109c268454010619aebb3
I20260810 03:50:58.466558 9296 ts_manager.cc:194] Registered new tserver with Master: 9950a0f607014fd18fa239f45369f36d (127.27.153.65:45569)
I20260810 03:50:58.467918 9296 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:35405
I20260810 03:50:58.477440 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:58.466529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:37893 (local address 127.27.153.65:55655)
0810 03:50:58.466833 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:58.466847 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:58.466925 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:58.469849 (+ 2924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:58.469852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:58.470085 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:58.470226 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:58.470235 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.471170 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.471173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:58.471990 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:58.472001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.472577 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.472580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:58.472687 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:58.473250 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:58.473270 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:58.473660 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:58.475736 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:58.475739 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:58.475742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:58.475971 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:58.476308 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:58.476311 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:58.476312 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:58.476413 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:58.476924 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:58.476926 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:58.477028 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 03:50:58.477141 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 03:50:58.477283 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":127,"threads_started":1}
I20260810 03:50:58.477512 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:58.469500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:55655 (local address 127.27.153.126:37893)
0810 03:50:58.469644 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:58.469648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:58.469663 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:50:58.469696 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:58.469698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:58.469734 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:58.469799 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:58.470370 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.471011 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.472118 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.472427 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.472541 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:58.473818 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:58.473823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:58.473825 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:58.473854 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:58.475612 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:58.476080 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:58.476083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:58.476085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:58.476186 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:58.476529 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:58.476532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:58.476534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:58.476799 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:58.476912 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:58.477217 (+ 305us) server_negotiation.cc:299] Negotiation successful
0810 03:50:58.477351 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
WARNING: no policy specified for oryx/127.27.153.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.153.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.153.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.66.keytab.
Entry for principal oryx/127.27.153.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.66.keytab.
I20260810 03:50:58.499199 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:37893
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.66.keytab
--principal=oryx/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:34857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:58.614398 9683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:58.614720 9683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:58.614787 9683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:58.619065 9683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:58.619153 9683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:58.619220 9683 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:50:58.619280 9683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:50:58.625082 9683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:34857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.66.keytab
--principal=oryx/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.153.126:37893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:58.626344 9683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:58.627429 9683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:58.638708 9700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:58.638669 9691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:58.638720 9696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:58.638900 9683 server_base.cc:1034] running on GCE node
I20260810 03:50:58.639580 9683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:58.640336 9683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:58.641543 9683 hybrid_clock.cc:648] HybridClock initialized: now 1786333858641492 us; error 63 us; skew 500 ppm
Aug 10 03:50:58 dist-test-slave-pww0 krb5kdc[9222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333858, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:58.644832 9683 init.cc:377] Logged in from keytab as oryx/127.27.153.66@KRBTEST.COM (short username oryx)
I20260810 03:50:58.645933 9683 webserver.cc:460] Webserver started at http://127.27.153.66:40495/ using document root <none> and password file <none>
I20260810 03:50:58.646571 9683 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:58.646642 9683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:58.646831 9683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:58.649188 9683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "d899c3a6763243fa8bf0376062da8249"
format_stamp: "Formatted at 2026-08-10 03:50:58 on dist-test-slave-pww0"
server_key: "d7783af5ebf2d966d537f37e095d5d14"
server_key_iv: "601f4970032cfd36c7165a3b8eeb67af"
server_key_version: "encryptionkey@0"
I20260810 03:50:58.649806 9683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "d899c3a6763243fa8bf0376062da8249"
format_stamp: "Formatted at 2026-08-10 03:50:58 on dist-test-slave-pww0"
server_key: "d7783af5ebf2d966d537f37e095d5d14"
server_key_iv: "601f4970032cfd36c7165a3b8eeb67af"
server_key_version: "encryptionkey@0"
I20260810 03:50:58.654167 9683 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:50:58.657271 9711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:58.658820 9683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:50:58.658964 9683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "d899c3a6763243fa8bf0376062da8249"
format_stamp: "Formatted at 2026-08-10 03:50:58 on dist-test-slave-pww0"
server_key: "d7783af5ebf2d966d537f37e095d5d14"
server_key_iv: "601f4970032cfd36c7165a3b8eeb67af"
server_key_version: "encryptionkey@0"
I20260810 03:50:58.659111 9683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:58.682214 9683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:58.685983 9683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:58.686189 9683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:58.687901 9683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:58.687978 9683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:58.688040 9683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:58.688097 9683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:50:58 dist-test-slave-pww0 krb5kdc[9222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333858, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM
I20260810 03:50:58.698776 9683 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:40937
I20260810 03:50:58.698882 9909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:40937 every 8 connection(s)
I20260810 03:50:58.699817 9683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:50:58.701916 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:58.688658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:44453 (local address 127.27.153.126:37893)
0810 03:50:58.688815 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:58.688820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:58.689798 (+ 978us) server_negotiation.cc:407] Connection header received
0810 03:50:58.690894 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:58.690898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:58.690964 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:58.691025 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:58.692719 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.693313 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.694457 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.694719 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.694820 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:58.698015 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:58.698019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:58.698021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:58.698051 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:58.699724 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:58.700306 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:58.700311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:58.700313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:58.700413 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:58.700793 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:58.700797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:58.700799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:58.701025 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:58.701116 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:58.701665 (+ 549us) server_negotiation.cc:299] Negotiation successful
0810 03:50:58.701785 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:50:58.702783 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:58.689137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:37893 (local address 127.27.153.66:44453)
0810 03:50:58.689614 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:58.689670 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:58.690631 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:58.691497 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:58.691508 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:58.691925 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:58.692559 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:58.692576 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.693438 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.693441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:58.694336 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:58.694346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.694879 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.694883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:58.695050 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:58.695732 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:58.695753 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:58.697820 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:58.699861 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:58.699868 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:58.699885 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:58.700175 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:58.700520 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:58.700524 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:58.700525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:58.700689 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:58.701136 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:58.701159 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:58.701422 (+ 263us) client_negotiation.cc:769] Sending connection context
0810 03:50:58.701582 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 03:50:58.701854 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":102,"threads_started":1}
I20260810 03:50:58.704144 9910 heartbeater.cc:347] Connected to a master server at 127.27.153.126:37893
I20260810 03:50:58.704294 9910 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:58.704941 9910 heartbeater.cc:511] Master 127.27.153.126:37893 requested a full tablet report, sending...
I20260810 03:50:58.706283 9296 ts_manager.cc:194] Registered new tserver with Master: d899c3a6763243fa8bf0376062da8249 (127.27.153.66:40937)
I20260810 03:50:58.707034 9296 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:44453
I20260810 03:50:58.707310 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 9683
I20260810 03:50:58.707440 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:50:58.707789 28261 external_mini_cluster.cc:1428] Setting key fd5210dfc1d8f34cff1dd9542377773e
I20260810 03:50:58.714754 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:58.706437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:35911 (local address 127.27.153.126:37893)
0810 03:50:58.706571 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:58.706576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:58.706823 (+ 247us) server_negotiation.cc:407] Connection header received
0810 03:50:58.706937 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:58.706939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:58.706980 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:58.707077 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:58.707929 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.708528 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.709707 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.710030 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.710133 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:58.711367 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:58.711371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:58.711373 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:58.711403 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:58.712881 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:58.713343 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:58.713346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:58.713349 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:58.713497 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:58.713820 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:58.713823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:58.713825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:58.714077 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:58.714171 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:58.714459 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 03:50:58.714584 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 03:50:58.714764 9918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:58.706375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:37893 (local address 127.27.153.66:35911)
0810 03:50:58.706719 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:58.706746 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:58.706860 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:58.707185 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:58.707189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:58.707613 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:58.707783 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:58.707791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.708710 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.708713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:58.709579 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:58.709590 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.710168 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.710171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:58.710248 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:58.710845 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:58.710858 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:58.711201 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:58.713000 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:58.713003 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:58.713005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:58.713236 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:58.713626 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:58.713629 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:58.713631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:58.713728 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:58.714176 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:58.714180 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:58.714298 (+ 118us) client_negotiation.cc:769] Sending connection context
0810 03:50:58.714388 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 03:50:58.714562 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for oryx/127.27.153.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.153.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.153.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.67.keytab.
Entry for principal oryx/127.27.153.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.67.keytab.
I20260810 03:50:58.738929 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:37893
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.67.keytab
--principal=oryx/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:34857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:50:58.860224 9924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:50:58.860523 9924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:50:58.860571 9924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:50:58.864845 9924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:50:58.864931 9924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:50:58.864974 9924 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 03:50:58.865013 9924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:50:58.870873 9924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:34857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.67.keytab
--principal=oryx/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.153.126:37893
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:50:58.872116 9924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:50:58.873184 9924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:50:58.883287 9930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:58.883580 9932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:50:58.883307 9939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:50:58.884081 9924 server_base.cc:1034] running on GCE node
I20260810 03:50:58.884631 9924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:50:58.885423 9924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:50:58.886626 9924 hybrid_clock.cc:648] HybridClock initialized: now 1786333858886592 us; error 48 us; skew 500 ppm
Aug 10 03:50:58 dist-test-slave-pww0 krb5kdc[9222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333858, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:50:58.889587 9924 init.cc:377] Logged in from keytab as oryx/127.27.153.67@KRBTEST.COM (short username oryx)
I20260810 03:50:58.890586 9924 webserver.cc:460] Webserver started at http://127.27.153.67:39549/ using document root <none> and password file <none>
I20260810 03:50:58.891155 9924 fs_manager.cc:359] Metadata directory not provided
I20260810 03:50:58.891228 9924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:50:58.891502 9924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:50:58.893679 9924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "9a3ee14cc82a4ad397efd5443ee068ba"
format_stamp: "Formatted at 2026-08-10 03:50:58 on dist-test-slave-pww0"
server_key: "c901e2917523f8b75f68a3cae08ad8fa"
server_key_iv: "32f3179914d63245f23845a33baa5662"
server_key_version: "encryptionkey@0"
I20260810 03:50:58.894232 9924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "9a3ee14cc82a4ad397efd5443ee068ba"
format_stamp: "Formatted at 2026-08-10 03:50:58 on dist-test-slave-pww0"
server_key: "c901e2917523f8b75f68a3cae08ad8fa"
server_key_iv: "32f3179914d63245f23845a33baa5662"
server_key_version: "encryptionkey@0"
I20260810 03:50:58.898273 9924 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:50:58.900951 9953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:58.902315 9924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:50:58.902437 9924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "9a3ee14cc82a4ad397efd5443ee068ba"
format_stamp: "Formatted at 2026-08-10 03:50:58 on dist-test-slave-pww0"
server_key: "c901e2917523f8b75f68a3cae08ad8fa"
server_key_iv: "32f3179914d63245f23845a33baa5662"
server_key_version: "encryptionkey@0"
I20260810 03:50:58.902544 9924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:50:58.919243 9924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:50:58.922644 9924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:50:58.922839 9924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:50:58.924484 9924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:50:58.924541 9924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:50:58.924607 9924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:50:58.924633 9924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 03:50:58 dist-test-slave-pww0 krb5kdc[9222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333858, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM
I20260810 03:50:58.935405 9924 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:33099
I20260810 03:50:58.935438 10152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:33099 every 8 connection(s)
I20260810 03:50:58.936580 9924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:50:58.936810 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 9924
I20260810 03:50:58.936930 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:50:58.937288 28261 external_mini_cluster.cc:1428] Setting key e32bc8bb5f09d29d754289e0caa0f2d0
I20260810 03:50:58.939188 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:58.925321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:46951 (local address 127.27.153.126:37893)
0810 03:50:58.925440 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:58.925444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:58.926658 (+ 1214us) server_negotiation.cc:407] Connection header received
0810 03:50:58.927919 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:58.927922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:58.927985 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:58.928071 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:58.929503 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.930199 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.931498 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.931799 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.931899 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:58.935126 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:58.935129 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:58.935131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:58.935161 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:58.936776 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:58.937410 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:58.937414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:58.937416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:58.937515 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:58.937987 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:58.937989 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:58.937991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:58.938245 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:58.938361 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:58.938957 (+ 596us) server_negotiation.cc:299] Negotiation successful
0810 03:50:58.939068 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 03:50:58.940128 9963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:58.925911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:37893 (local address 127.27.153.67:46951)
0810 03:50:58.926480 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:58.926534 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:58.927666 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:58.928346 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:58.928355 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:58.928745 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:58.929335 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:58.929351 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.930338 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.930342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:58.931343 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:58.931355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.931970 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.931974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:58.932149 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:58.933367 (+ 1218us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:58.933389 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:58.934943 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:58.936916 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:58.936922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:58.936938 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:58.937277 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:58.937705 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:58.937708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:58.937710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:58.937881 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:58.938382 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:58.938388 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:58.938686 (+ 298us) client_negotiation.cc:769] Sending connection context
0810 03:50:58.938910 (+ 224us) client_negotiation.cc:240] Negotiation successful
0810 03:50:58.939256 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":135,"threads_started":1}
I20260810 03:50:58.941216 10156 heartbeater.cc:347] Connected to a master server at 127.27.153.126:37893
I20260810 03:50:58.941362 10156 heartbeater.cc:464] Registering TS with master...
I20260810 03:50:58.941943 10156 heartbeater.cc:511] Master 127.27.153.126:37893 requested a full tablet report, sending...
I20260810 03:50:58.943030 9297 ts_manager.cc:194] Registered new tserver with Master: 9a3ee14cc82a4ad397efd5443ee068ba (127.27.153.67:33099)
I20260810 03:50:58.943848 9297 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:46951
I20260810 03:50:58.949992 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:58.943572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:37893 (local address 127.27.153.67:53463)
0810 03:50:58.943906 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:58.943918 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:58.944058 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:58.944303 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:58.944305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:58.944510 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:58.944629 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:58.944636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.945364 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.945367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:58.946105 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:58.946115 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.946483 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.946485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:58.946549 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:58.947030 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:58.947042 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:58.947381 (+ 339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:58.948786 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:58.948789 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:58.948790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:58.948972 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:58.949172 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:58.949174 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:58.949175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:58.949254 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:58.949544 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:58.949546 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:58.949620 (+ 74us) client_negotiation.cc:769] Sending connection context
0810 03:50:58.949695 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 03:50:58.949812 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":108,"threads_started":1}
I20260810 03:50:58.950091 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:58.943630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:53463 (local address 127.27.153.126:37893)
0810 03:50:58.943771 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:58.943776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:58.943987 (+ 211us) server_negotiation.cc:407] Connection header received
0810 03:50:58.944142 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:58.944145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:58.944189 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:58.944267 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:58.944750 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.945240 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.946194 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.946406 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.946473 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:58.947554 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:58.947559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:58.947562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:58.947582 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:58.948691 (+ 1109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:58.949044 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:58.949046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:58.949047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:58.949113 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:58.949323 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:58.949324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:58.949325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:58.949481 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:58.949551 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:58.949810 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 03:50:58.949940 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 03:50:58.951422 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 03:50:58 dist-test-slave-pww0 krb5kdc[9222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333858, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 03:50:58 dist-test-slave-pww0 krb5kdc[9222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:50:58 dist-test-slave-pww0 krb5kdc[9222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 03:50:58.977751 9387 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:50:58.971342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39398 (local address 127.27.153.126:37893)
0810 03:50:58.971523 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:58.971527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:58.971541 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:50:58.971587 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:58.971589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:58.971633 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:58.971707 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:58.972494 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.973047 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.973894 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.974119 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.974199 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:58.977578 (+ 3379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39398: BlockingRecv error: recv got EOF from 127.0.0.1:39398 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 03:50:58 dist-test-slave-pww0 krb5kdc[9222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333858, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM
I20260810 03:50:58.987319 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:58.979188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39406 (local address 127.27.153.126:37893)
0810 03:50:58.979350 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:58.979353 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:58.979395 (+ 42us) server_negotiation.cc:407] Connection header received
0810 03:50:58.979565 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:58.979568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:58.979612 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:58.979679 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:58.980290 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.980796 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.981578 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:58.981794 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:58.981884 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:58.984420 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:58.984424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:58.984426 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:58.984458 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:58.985673 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:58.986083 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:58.986086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:58.986100 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:58.986186 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:58.986456 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:58.986459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:58.986461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:58.986690 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:58.986814 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:58.987056 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 03:50:58.987173 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 03:50:58.990437 9297 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:50:58.993675 9297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:50:59.009078 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.002443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:33099 (local address 127.0.0.1:52442)
0810 03:50:59.003115 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:59.003171 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:59.003313 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:59.003970 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:59.003977 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:59.004005 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:59.004168 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.004175 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.005625 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.005628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.007483 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.007496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.008484 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.008487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.008607 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.008652 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 03:50:59.008775 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 03:50:59.008921 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":54,"thread_start_us":104,"threads_started":1}
I20260810 03:50:59.009088 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.002590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52442 (local address 127.27.153.67:33099)
0810 03:50:59.003251 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:59.003263 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:59.003300 (+ 37us) server_negotiation.cc:407] Connection header received
0810 03:50:59.003514 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:59.003520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:59.003758 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:59.004359 (+ 601us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:59.004514 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.005525 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.007639 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.008363 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.008468 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.008657 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:59.008820 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 03:50:59.008921 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":111,"threads_started":1}
I20260810 03:50:59.009366 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.002517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:45569 (local address 127.0.0.1:52052)
0810 03:50:59.003813 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:59.003827 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:59.003911 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:59.004552 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:59.004555 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:59.004580 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:59.004686 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.004693 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.006085 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.006089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.007487 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.007496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.008824 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.008828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.008993 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.009059 (+ 66us) client_negotiation.cc:769] Sending connection context
0810 03:50:59.009129 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 03:50:59.009209 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1229,"mutex_wait_us":121,"thread_start_us":117,"threads_started":1}
I20260810 03:50:59.009866 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.003306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52052 (local address 127.27.153.65:45569)
0810 03:50:59.003879 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:59.003886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:59.003909 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:50:59.004082 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:59.004088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:59.004292 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:59.004430 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:59.004847 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.005970 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.007641 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.008496 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.008606 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.008755 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:59.009606 (+ 851us) server_negotiation.cc:299] Negotiation successful
0810 03:50:59.009722 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":91,"threads_started":1}
I20260810 03:50:59.012739 10038 tablet_service.cc:1467] Processing CreateTablet for tablet b3dae05280f5487f8a67465474302834 (DEFAULT_TABLE table=test-table [id=261f859a0ea34bac82a547e746a45e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:59.012739 9558 tablet_service.cc:1467] Processing CreateTablet for tablet b3dae05280f5487f8a67465474302834 (DEFAULT_TABLE table=test-table [id=261f859a0ea34bac82a547e746a45e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:59.014030 10038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3dae05280f5487f8a67465474302834. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:59.014030 9558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3dae05280f5487f8a67465474302834. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:59.019142 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.002736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:40937 (local address 127.0.0.1:36092)
0810 03:50:59.004270 (+ 1534us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:59.004285 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:59.004504 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:59.005027 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:59.005030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:59.005051 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:59.005161 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.005168 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.006374 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.006377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.007608 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.007621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.008831 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.008834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.009017 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.009063 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 03:50:59.009129 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 03:50:59.018957 (+ 9828us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1475,"mutex_wait_us":127,"thread_start_us":85,"threads_started":1}
I20260810 03:50:59.019304 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.003306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36092 (local address 127.27.153.66:40937)
0810 03:50:59.004131 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:59.004142 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:59.004452 (+ 310us) server_negotiation.cc:407] Connection header received
0810 03:50:59.004659 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:59.004664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:59.004878 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:59.005020 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:59.005271 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.006268 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.007715 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.008496 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.008606 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.008770 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:59.009682 (+ 912us) server_negotiation.cc:299] Negotiation successful
0810 03:50:59.009775 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":105,"threads_started":1}
I20260810 03:50:59.022292 10210 tablet_bootstrap.cc:492] T b3dae05280f5487f8a67465474302834 P 9950a0f607014fd18fa239f45369f36d: Bootstrap starting.
I20260810 03:50:59.022612 9798 tablet_service.cc:1467] Processing CreateTablet for tablet b3dae05280f5487f8a67465474302834 (DEFAULT_TABLE table=test-table [id=261f859a0ea34bac82a547e746a45e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:59.023720 10213 tablet_bootstrap.cc:492] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba: Bootstrap starting.
I20260810 03:50:59.023869 9798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3dae05280f5487f8a67465474302834. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:59.024842 10210 tablet_bootstrap.cc:654] T b3dae05280f5487f8a67465474302834 P 9950a0f607014fd18fa239f45369f36d: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:59.025660 10213 tablet_bootstrap.cc:654] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:59.025739 10210 log.cc:824] T b3dae05280f5487f8a67465474302834 P 9950a0f607014fd18fa239f45369f36d: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:59.026460 10213 log.cc:824] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:59.028256 10210 tablet_bootstrap.cc:492] T b3dae05280f5487f8a67465474302834 P 9950a0f607014fd18fa239f45369f36d: No bootstrap required, opened a new log
I20260810 03:50:59.028337 10213 tablet_bootstrap.cc:492] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba: No bootstrap required, opened a new log
I20260810 03:50:59.028499 10210 ts_tablet_manager.cc:1397] T b3dae05280f5487f8a67465474302834 P 9950a0f607014fd18fa239f45369f36d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:50:59.028537 10213 ts_tablet_manager.cc:1397] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 03:50:59.031853 10213 raft_consensus.cc:348] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } } peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } }
I20260810 03:50:59.032204 10213 raft_consensus.cc:374] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:59.032266 10213 raft_consensus.cc:729] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3ee14cc82a4ad397efd5443ee068ba, State: Initialized, Role: FOLLOWER
I20260810 03:50:59.032814 10213 consensus_queue.cc:260] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } } peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } }
I20260810 03:50:59.033427 10216 tablet_bootstrap.cc:492] T b3dae05280f5487f8a67465474302834 P d899c3a6763243fa8bf0376062da8249: Bootstrap starting.
I20260810 03:50:59.033274 10210 raft_consensus.cc:348] T b3dae05280f5487f8a67465474302834 P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } } peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } }
I20260810 03:50:59.033547 10210 raft_consensus.cc:374] T b3dae05280f5487f8a67465474302834 P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:59.033604 10210 raft_consensus.cc:729] T b3dae05280f5487f8a67465474302834 P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9950a0f607014fd18fa239f45369f36d, State: Initialized, Role: FOLLOWER
I20260810 03:50:59.033993 10213 ts_tablet_manager.cc:1428] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 03:50:59.034058 10210 consensus_queue.cc:260] T b3dae05280f5487f8a67465474302834 P 9950a0f607014fd18fa239f45369f36d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } } peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } }
I20260810 03:50:59.034389 10156 heartbeater.cc:503] Master 127.27.153.126:37893 was elected leader, sending a full tablet report...
I20260810 03:50:59.034839 9666 heartbeater.cc:503] Master 127.27.153.126:37893 was elected leader, sending a full tablet report...
I20260810 03:50:59.035072 10210 ts_tablet_manager.cc:1428] T b3dae05280f5487f8a67465474302834 P 9950a0f607014fd18fa239f45369f36d: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 03:50:59.037074 10216 tablet_bootstrap.cc:654] T b3dae05280f5487f8a67465474302834 P d899c3a6763243fa8bf0376062da8249: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:59.038061 10216 log.cc:824] T b3dae05280f5487f8a67465474302834 P d899c3a6763243fa8bf0376062da8249: Log is configured to *not* fsync() on all Append() calls
I20260810 03:50:59.040311 10216 tablet_bootstrap.cc:492] T b3dae05280f5487f8a67465474302834 P d899c3a6763243fa8bf0376062da8249: No bootstrap required, opened a new log
I20260810 03:50:59.040510 10216 ts_tablet_manager.cc:1397] T b3dae05280f5487f8a67465474302834 P d899c3a6763243fa8bf0376062da8249: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 03:50:59.043856 10216 raft_consensus.cc:348] T b3dae05280f5487f8a67465474302834 P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } } peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } }
I20260810 03:50:59.044092 10216 raft_consensus.cc:374] T b3dae05280f5487f8a67465474302834 P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:59.044143 10216 raft_consensus.cc:729] T b3dae05280f5487f8a67465474302834 P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d899c3a6763243fa8bf0376062da8249, State: Initialized, Role: FOLLOWER
I20260810 03:50:59.044584 10216 consensus_queue.cc:260] T b3dae05280f5487f8a67465474302834 P d899c3a6763243fa8bf0376062da8249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } } peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } }
I20260810 03:50:59.045266 9910 heartbeater.cc:503] Master 127.27.153.126:37893 was elected leader, sending a full tablet report...
I20260810 03:50:59.045609 10216 ts_tablet_manager.cc:1428] T b3dae05280f5487f8a67465474302834 P d899c3a6763243fa8bf0376062da8249: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 03:50:59.085433 10219 raft_consensus.cc:482] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:50:59.085687 10219 raft_consensus.cc:504] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } } peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } }
I20260810 03:50:59.086843 10219 leader_election.cc:288] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9950a0f607014fd18fa239f45369f36d (127.27.153.65:45569), d899c3a6763243fa8bf0376062da8249 (127.27.153.66:40937)
I20260810 03:50:59.090744 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.087500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:54825 (local address 127.27.153.65:45569)
0810 03:50:59.087649 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:59.087653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:59.087668 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:50:59.087724 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:59.087727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:59.087768 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:59.087831 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:59.088160 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.088690 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.089754 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.090173 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.090284 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.090355 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:59.090544 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 03:50:59.090627 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:50:59.090744 9963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.087086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:45569 (local address 127.27.153.67:54825)
0810 03:50:59.087268 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:59.087284 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:59.087382 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:59.087850 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:59.087853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:59.087871 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:59.088050 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.088056 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.088790 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.088794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.089612 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.089622 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.090284 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.090288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.090432 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.090451 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 03:50:59.090519 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 03:50:59.090599 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":36,"spinlock_wait_cycles":8960}
I20260810 03:50:59.090974 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.087234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:37135 (local address 127.27.153.66:40937)
0810 03:50:59.087372 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:59.087377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:59.087607 (+ 230us) server_negotiation.cc:407] Connection header received
0810 03:50:59.087736 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:59.087738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:59.087790 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:59.087855 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:59.088462 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.088984 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.090052 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.090470 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.090582 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.090634 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:59.090795 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 03:50:59.090869 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 03:50:59.090988 10223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.087101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:40937 (local address 127.27.153.67:37135)
0810 03:50:59.087531 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:59.087543 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:59.087643 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:59.087886 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:59.087889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:59.087933 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:59.088344 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.088351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.089087 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.089090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.089924 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.089935 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.090582 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.090586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.090682 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.090700 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 03:50:59.090765 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 03:50:59.090833 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":49,"spinlock_wait_cycles":22656,"thread_start_us":111,"threads_started":1}
I20260810 03:50:59.091547 9597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3dae05280f5487f8a67465474302834" candidate_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9950a0f607014fd18fa239f45369f36d" is_pre_election: true
I20260810 03:50:59.091574 9827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3dae05280f5487f8a67465474302834" candidate_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d899c3a6763243fa8bf0376062da8249" is_pre_election: true
I20260810 03:50:59.091904 9827 raft_consensus.cc:2454] T b3dae05280f5487f8a67465474302834 P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3ee14cc82a4ad397efd5443ee068ba in term 0.
I20260810 03:50:59.091904 9597 raft_consensus.cc:2454] T b3dae05280f5487f8a67465474302834 P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3ee14cc82a4ad397efd5443ee068ba in term 0.
I20260810 03:50:59.092442 9956 leader_election.cc:302] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3ee14cc82a4ad397efd5443ee068ba, d899c3a6763243fa8bf0376062da8249; no voters:
I20260810 03:50:59.092729 10219 raft_consensus.cc:2781] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:50:59.092828 10219 raft_consensus.cc:482] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:50:59.092897 10219 raft_consensus.cc:3037] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:59.094063 10219 raft_consensus.cc:504] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } } peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } }
I20260810 03:50:59.094457 10219 leader_election.cc:288] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba [CANDIDATE]: Term 1 election: Requested vote from peers 9950a0f607014fd18fa239f45369f36d (127.27.153.65:45569), d899c3a6763243fa8bf0376062da8249 (127.27.153.66:40937)
I20260810 03:50:59.094851 9827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3dae05280f5487f8a67465474302834" candidate_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d899c3a6763243fa8bf0376062da8249"
I20260810 03:50:59.094843 9597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3dae05280f5487f8a67465474302834" candidate_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9950a0f607014fd18fa239f45369f36d"
I20260810 03:50:59.094985 9827 raft_consensus.cc:3037] T b3dae05280f5487f8a67465474302834 P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:59.094985 9597 raft_consensus.cc:3037] T b3dae05280f5487f8a67465474302834 P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:59.096377 9827 raft_consensus.cc:2454] T b3dae05280f5487f8a67465474302834 P d899c3a6763243fa8bf0376062da8249 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3ee14cc82a4ad397efd5443ee068ba in term 1.
I20260810 03:50:59.096377 9597 raft_consensus.cc:2454] T b3dae05280f5487f8a67465474302834 P 9950a0f607014fd18fa239f45369f36d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3ee14cc82a4ad397efd5443ee068ba in term 1.
I20260810 03:50:59.096733 9954 leader_election.cc:302] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9950a0f607014fd18fa239f45369f36d, 9a3ee14cc82a4ad397efd5443ee068ba; no voters:
I20260810 03:50:59.096941 10219 raft_consensus.cc:2781] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:59.097177 10219 raft_consensus.cc:686] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba [term 1 LEADER]: Becoming Leader. State: Replica: 9a3ee14cc82a4ad397efd5443ee068ba, State: Running, Role: LEADER
I20260810 03:50:59.097549 10219 consensus_queue.cc:237] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } } peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } }
I20260810 03:50:59.100807 9297 catalog_manager.cc:5495] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3ee14cc82a4ad397efd5443ee068ba (127.27.153.67). New cstate: current_term: 1 leader_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } health_report { overall_health: UNKNOWN } } }
I20260810 03:50:59.140771 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.132689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52446 (local address 127.27.153.67:33099)
0810 03:50:59.132837 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:59.132843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:59.132863 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:50:59.133054 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:59.133058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:59.133152 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:59.133291 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:50:59.135523 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.136557 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.138630 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.138954 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.139058 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.139382 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:50:59.140069 (+ 687us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:50:59.140185 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:59.140515 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 03:50:59.140592 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 03:50:59.152173 9597 raft_consensus.cc:1261] T b3dae05280f5487f8a67465474302834 P 9950a0f607014fd18fa239f45369f36d [term 1 FOLLOWER]: Refusing update from remote peer 9a3ee14cc82a4ad397efd5443ee068ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:59.152257 9827 raft_consensus.cc:1261] T b3dae05280f5487f8a67465474302834 P d899c3a6763243fa8bf0376062da8249 [term 1 FOLLOWER]: Refusing update from remote peer 9a3ee14cc82a4ad397efd5443ee068ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:59.153038 10225 consensus_queue.cc:1035] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:59.153244 10219 consensus_queue.cc:1035] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:59.165519 10232 mvcc.cc:205] Tried to move back new op lower bound from 7316823487062949888 to 7316823486866419712. Current Snapshot: MvccSnapshot[applied={T|T < 7316823487062949888}]
I20260810 03:50:59.173301 9297 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:59.173503 9297 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:59.173578 10233 mvcc.cc:205] Tried to move back new op lower bound from 7316823487062949888 to 7316823486866419712. Current Snapshot: MvccSnapshot[applied={T|T < 7316823487062949888}]
I20260810 03:50:59.176219 9297 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94: Sending DeleteTablet for 3 replicas of tablet b3dae05280f5487f8a67465474302834
I20260810 03:50:59.177132 9558 tablet_service.cc:1514] Processing DeleteTablet for tablet b3dae05280f5487f8a67465474302834 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:59 UTC) from {username='oryx', principal='oryx/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:52052
I20260810 03:50:59.177137 10038 tablet_service.cc:1514] Processing DeleteTablet for tablet b3dae05280f5487f8a67465474302834 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:59 UTC) from {username='oryx', principal='oryx/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:52442
I20260810 03:50:59.178176 9798 tablet_service.cc:1514] Processing DeleteTablet for tablet b3dae05280f5487f8a67465474302834 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:59 UTC) from {username='oryx', principal='oryx/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:36092
I20260810 03:50:59.178694 9297 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:50:59.179286 9297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:50:59.181316 10256 tablet_replica.cc:330] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba: stopping tablet replica
I20260810 03:50:59.181705 10256 raft_consensus.cc:2229] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:50:59.182123 10256 raft_consensus.cc:2258] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:59.182554 10252 tablet_replica.cc:330] T b3dae05280f5487f8a67465474302834 P 9950a0f607014fd18fa239f45369f36d: stopping tablet replica
I20260810 03:50:59.182960 10252 raft_consensus.cc:2229] T b3dae05280f5487f8a67465474302834 P 9950a0f607014fd18fa239f45369f36d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:59.183369 10252 raft_consensus.cc:2258] T b3dae05280f5487f8a67465474302834 P 9950a0f607014fd18fa239f45369f36d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:59.185041 10038 tablet_service.cc:1467] Processing CreateTablet for tablet f5f4d2dbda604158b7e4a3bc8eccbe59 (DEFAULT_TABLE table=test-table [id=07502a0bf34344b19bcde02d2c18a6ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:59.185308 10038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5f4d2dbda604158b7e4a3bc8eccbe59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:59.187427 9798 tablet_service.cc:1467] Processing CreateTablet for tablet f5f4d2dbda604158b7e4a3bc8eccbe59 (DEFAULT_TABLE table=test-table [id=07502a0bf34344b19bcde02d2c18a6ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:59.187615 9558 tablet_service.cc:1467] Processing CreateTablet for tablet f5f4d2dbda604158b7e4a3bc8eccbe59 (DEFAULT_TABLE table=test-table [id=07502a0bf34344b19bcde02d2c18a6ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:59.187777 9798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5f4d2dbda604158b7e4a3bc8eccbe59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:59.187835 10256 ts_tablet_manager.cc:1905] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:59.187884 9558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5f4d2dbda604158b7e4a3bc8eccbe59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:59.187898 10213 tablet_bootstrap.cc:492] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9a3ee14cc82a4ad397efd5443ee068ba: Bootstrap starting.
I20260810 03:50:59.189034 10213 tablet_bootstrap.cc:654] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9a3ee14cc82a4ad397efd5443ee068ba: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:59.190275 10210 tablet_bootstrap.cc:492] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9950a0f607014fd18fa239f45369f36d: Bootstrap starting.
I20260810 03:50:59.190407 10216 tablet_bootstrap.cc:492] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249: Bootstrap starting.
I20260810 03:50:59.190414 10256 ts_tablet_manager.cc:1918] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:50:59.190510 10256 log.cc:1196] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/b3dae05280f5487f8a67465474302834
I20260810 03:50:59.190889 10256 ts_tablet_manager.cc:1939] T b3dae05280f5487f8a67465474302834 P 9a3ee14cc82a4ad397efd5443ee068ba: Deleting consensus metadata
I20260810 03:50:59.191263 10210 tablet_bootstrap.cc:654] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9950a0f607014fd18fa239f45369f36d: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:59.191550 10216 tablet_bootstrap.cc:654] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:59.192318 9277 catalog_manager.cc:4841] TS 9a3ee14cc82a4ad397efd5443ee068ba (127.27.153.67:33099): tablet b3dae05280f5487f8a67465474302834 (table test-table [id=261f859a0ea34bac82a547e746a45e37]) successfully deleted
I20260810 03:50:59.192587 10257 tablet_replica.cc:330] T b3dae05280f5487f8a67465474302834 P d899c3a6763243fa8bf0376062da8249: stopping tablet replica
I20260810 03:50:59.192906 10257 raft_consensus.cc:2229] T b3dae05280f5487f8a67465474302834 P d899c3a6763243fa8bf0376062da8249 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:59.193156 10252 ts_tablet_manager.cc:1905] T b3dae05280f5487f8a67465474302834 P 9950a0f607014fd18fa239f45369f36d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:59.193221 10257 raft_consensus.cc:2258] T b3dae05280f5487f8a67465474302834 P d899c3a6763243fa8bf0376062da8249 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:59.194947 10257 ts_tablet_manager.cc:1905] T b3dae05280f5487f8a67465474302834 P d899c3a6763243fa8bf0376062da8249: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:59.196167 10252 ts_tablet_manager.cc:1918] T b3dae05280f5487f8a67465474302834 P 9950a0f607014fd18fa239f45369f36d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:50:59.196314 10252 log.cc:1196] T b3dae05280f5487f8a67465474302834 P 9950a0f607014fd18fa239f45369f36d: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-0/wal/wals/b3dae05280f5487f8a67465474302834
I20260810 03:50:59.197230 10252 ts_tablet_manager.cc:1939] T b3dae05280f5487f8a67465474302834 P 9950a0f607014fd18fa239f45369f36d: Deleting consensus metadata
I20260810 03:50:59.197937 10257 ts_tablet_manager.cc:1918] T b3dae05280f5487f8a67465474302834 P d899c3a6763243fa8bf0376062da8249: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:50:59.198037 10257 log.cc:1196] T b3dae05280f5487f8a67465474302834 P d899c3a6763243fa8bf0376062da8249: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/b3dae05280f5487f8a67465474302834
I20260810 03:50:59.198369 10257 ts_tablet_manager.cc:1939] T b3dae05280f5487f8a67465474302834 P d899c3a6763243fa8bf0376062da8249: Deleting consensus metadata
I20260810 03:50:59.199088 9275 catalog_manager.cc:4841] TS 9950a0f607014fd18fa239f45369f36d (127.27.153.65:45569): tablet b3dae05280f5487f8a67465474302834 (table test-table [id=261f859a0ea34bac82a547e746a45e37]) successfully deleted
I20260810 03:50:59.199918 9278 catalog_manager.cc:4841] TS d899c3a6763243fa8bf0376062da8249 (127.27.153.66:40937): tablet b3dae05280f5487f8a67465474302834 (table test-table [id=261f859a0ea34bac82a547e746a45e37]) successfully deleted
I20260810 03:50:59.200423 10216 tablet_bootstrap.cc:492] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249: No bootstrap required, opened a new log
I20260810 03:50:59.200539 10216 ts_tablet_manager.cc:1397] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260810 03:50:59.200927 10213 tablet_bootstrap.cc:492] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9a3ee14cc82a4ad397efd5443ee068ba: No bootstrap required, opened a new log
I20260810 03:50:59.201032 10213 ts_tablet_manager.cc:1397] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9a3ee14cc82a4ad397efd5443ee068ba: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260810 03:50:59.201249 10210 tablet_bootstrap.cc:492] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9950a0f607014fd18fa239f45369f36d: No bootstrap required, opened a new log
I20260810 03:50:59.201335 10210 ts_tablet_manager.cc:1397] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9950a0f607014fd18fa239f45369f36d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260810 03:50:59.201542 10213 raft_consensus.cc:348] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.201663 10213 raft_consensus.cc:374] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:59.201694 10213 raft_consensus.cc:729] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3ee14cc82a4ad397efd5443ee068ba, State: Initialized, Role: FOLLOWER
I20260810 03:50:59.201156 10216 raft_consensus.cc:348] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.201776 10210 raft_consensus.cc:348] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.201825 10213 consensus_queue.cc:260] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9a3ee14cc82a4ad397efd5443ee068ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.201876 10210 raft_consensus.cc:374] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:59.201877 10216 raft_consensus.cc:374] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:59.201908 10210 raft_consensus.cc:729] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9950a0f607014fd18fa239f45369f36d, State: Initialized, Role: FOLLOWER
I20260810 03:50:59.201929 10216 raft_consensus.cc:729] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d899c3a6763243fa8bf0376062da8249, State: Initialized, Role: FOLLOWER
I20260810 03:50:59.202023 10210 consensus_queue.cc:260] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9950a0f607014fd18fa239f45369f36d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.202101 10213 ts_tablet_manager.cc:1428] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9a3ee14cc82a4ad397efd5443ee068ba: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:50:59.202088 10216 consensus_queue.cc:260] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.202275 10210 ts_tablet_manager.cc:1428] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9950a0f607014fd18fa239f45369f36d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 03:50:59.202385 10216 ts_tablet_manager.cc:1428] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260810 03:50:59.210497 9667 tablet.cc:2369] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9950a0f607014fd18fa239f45369f36d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:50:59.262923 10222 raft_consensus.cc:482] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:50:59.263082 10222 raft_consensus.cc:504] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.264430 10222 leader_election.cc:288] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9950a0f607014fd18fa239f45369f36d (127.27.153.65:45569), 9a3ee14cc82a4ad397efd5443ee068ba (127.27.153.67:33099)
I20260810 03:50:59.268884 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.265204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:60433 (local address 127.27.153.67:33099)
0810 03:50:59.265340 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:59.265345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:59.265359 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:50:59.265415 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:59.265418 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:59.265469 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:59.265546 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:59.265926 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.266646 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.267826 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.268304 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.268434 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.268493 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:59.268673 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 03:50:59.268752 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 03:50:59.268903 10267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.264725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:33099 (local address 127.27.153.66:60433)
0810 03:50:59.265110 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:59.265123 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:59.265194 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:59.265660 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:59.265664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:59.265683 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:59.265793 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.265799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.266761 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.266765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.267686 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.267699 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.268431 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.268435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.268544 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.268572 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 03:50:59.268643 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 03:50:59.268723 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":28,"spinlock_wait_cycles":2944,"thread_start_us":80,"threads_started":1}
I20260810 03:50:59.269400 10263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.264636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:45569 (local address 127.27.153.66:34339)
0810 03:50:59.265012 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:59.265028 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:59.265192 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:59.265504 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:59.265506 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:59.265525 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:59.265637 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.265642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.266638 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.266641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.268020 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.268033 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.268954 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.268958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.269051 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.269076 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 03:50:59.269182 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 03:50:59.269262 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":43,"thread_start_us":100,"threads_started":1}
I20260810 03:50:59.269433 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.264747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:34339 (local address 127.27.153.65:45569)
0810 03:50:59.264876 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:59.264881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:59.265102 (+ 221us) server_negotiation.cc:407] Connection header received
0810 03:50:59.265361 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:59.265364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:59.265410 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:59.265472 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:59.265757 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.266518 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.268164 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.268724 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.268841 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.268888 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:59.269234 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 03:50:59.269325 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 03:50:59.269886 9597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f4d2dbda604158b7e4a3bc8eccbe59" candidate_uuid: "d899c3a6763243fa8bf0376062da8249" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9950a0f607014fd18fa239f45369f36d" is_pre_election: true
I20260810 03:50:59.270044 9597 raft_consensus.cc:2454] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d899c3a6763243fa8bf0376062da8249 in term 0.
I20260810 03:50:59.270115 10075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f4d2dbda604158b7e4a3bc8eccbe59" candidate_uuid: "d899c3a6763243fa8bf0376062da8249" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" is_pre_election: true
I20260810 03:50:59.270365 10075 raft_consensus.cc:2454] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d899c3a6763243fa8bf0376062da8249 in term 0.
I20260810 03:50:59.270438 9714 leader_election.cc:302] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9950a0f607014fd18fa239f45369f36d, d899c3a6763243fa8bf0376062da8249; no voters:
I20260810 03:50:59.270751 10222 raft_consensus.cc:2781] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:50:59.270805 10222 raft_consensus.cc:482] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:50:59.270843 10222 raft_consensus.cc:3037] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:59.271893 10222 raft_consensus.cc:504] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.272262 10222 leader_election.cc:288] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249 [CANDIDATE]: Term 1 election: Requested vote from peers 9950a0f607014fd18fa239f45369f36d (127.27.153.65:45569), 9a3ee14cc82a4ad397efd5443ee068ba (127.27.153.67:33099)
I20260810 03:50:59.272678 9597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f4d2dbda604158b7e4a3bc8eccbe59" candidate_uuid: "d899c3a6763243fa8bf0376062da8249" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9950a0f607014fd18fa239f45369f36d"
I20260810 03:50:59.272811 9597 raft_consensus.cc:3037] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:59.274034 9597 raft_consensus.cc:2454] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9950a0f607014fd18fa239f45369f36d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d899c3a6763243fa8bf0376062da8249 in term 1.
I20260810 03:50:59.274379 9714 leader_election.cc:302] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9950a0f607014fd18fa239f45369f36d, d899c3a6763243fa8bf0376062da8249; no voters:
I20260810 03:50:59.274410 10075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5f4d2dbda604158b7e4a3bc8eccbe59" candidate_uuid: "d899c3a6763243fa8bf0376062da8249" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3ee14cc82a4ad397efd5443ee068ba"
I20260810 03:50:59.274575 10075 raft_consensus.cc:3037] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:59.274572 10222 raft_consensus.cc:2781] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:59.274830 10222 raft_consensus.cc:686] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249 [term 1 LEADER]: Becoming Leader. State: Replica: d899c3a6763243fa8bf0376062da8249, State: Running, Role: LEADER
I20260810 03:50:59.275111 10222 consensus_queue.cc:237] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.275848 10075 raft_consensus.cc:2454] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9a3ee14cc82a4ad397efd5443ee068ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d899c3a6763243fa8bf0376062da8249 in term 1.
I20260810 03:50:59.277755 9297 catalog_manager.cc:5495] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249 reported cstate change: term changed from 0 to 1, leader changed from <none> to d899c3a6763243fa8bf0376062da8249 (127.27.153.66). New cstate: current_term: 1 leader_uuid: "d899c3a6763243fa8bf0376062da8249" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } health_report { overall_health: UNKNOWN } } }
Aug 10 03:50:59 dist-test-slave-pww0 krb5kdc[9222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333858, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.126@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM
I20260810 03:50:59.326733 10277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.317401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:37893 (local address 127.0.0.1:39412)
0810 03:50:59.317751 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:59.317768 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:59.317889 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:59.318224 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:59.318226 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:59.318634 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:50:59.318733 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.318746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.319912 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.319915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.320490 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.320498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.321092 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.321095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.321180 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.321781 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:50:59.321793 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:50:59.323256 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:50:59.325165 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:59.325176 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:59.325182 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:59.325433 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:59.325685 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:50:59.325687 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:50:59.325688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:50:59.325786 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:50:59.326189 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:50:59.326195 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:50:59.326317 (+ 122us) client_negotiation.cc:769] Sending connection context
0810 03:50:59.326412 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 03:50:59.326567 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":3,"thread_start_us":132,"threads_started":1}
I20260810 03:50:59.326831 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.317533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39412 (local address 127.27.153.126:37893)
0810 03:50:59.317698 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:59.317703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:59.317851 (+ 148us) server_negotiation.cc:407] Connection header received
0810 03:50:59.317979 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:59.317982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:59.318042 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:59.318160 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:59.318877 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.319771 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.320623 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.320970 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.321086 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.323420 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:59.323424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:59.323427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:59.323485 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:59.325046 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:59.325521 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:59.325524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:59.325526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:59.325611 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:59.325856 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:59.325859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:59.325860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:59.326082 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:59.326172 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:59.326471 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 03:50:59.326639 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":63}
I20260810 03:50:59.331485 9297 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:39412:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 03:50:59.332413 9297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:50:59.338097 9558 tablet_service.cc:1467] Processing CreateTablet for tablet ae57381d7cdc40f0b02935bcea0f0d84 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b3f7218415ac44dcaec5446d6931bbd5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:50:59.338172 9798 tablet_service.cc:1467] Processing CreateTablet for tablet ae57381d7cdc40f0b02935bcea0f0d84 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b3f7218415ac44dcaec5446d6931bbd5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:50:59.338603 9798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae57381d7cdc40f0b02935bcea0f0d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:59.338605 9558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae57381d7cdc40f0b02935bcea0f0d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:59.339929 10038 tablet_service.cc:1467] Processing CreateTablet for tablet ae57381d7cdc40f0b02935bcea0f0d84 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b3f7218415ac44dcaec5446d6931bbd5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 03:50:59.340323 10038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae57381d7cdc40f0b02935bcea0f0d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:59.341606 10216 tablet_bootstrap.cc:492] T ae57381d7cdc40f0b02935bcea0f0d84 P d899c3a6763243fa8bf0376062da8249: Bootstrap starting.
I20260810 03:50:59.341797 10210 tablet_bootstrap.cc:492] T ae57381d7cdc40f0b02935bcea0f0d84 P 9950a0f607014fd18fa239f45369f36d: Bootstrap starting.
I20260810 03:50:59.342635 10216 tablet_bootstrap.cc:654] T ae57381d7cdc40f0b02935bcea0f0d84 P d899c3a6763243fa8bf0376062da8249: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:59.342777 10210 tablet_bootstrap.cc:654] T ae57381d7cdc40f0b02935bcea0f0d84 P 9950a0f607014fd18fa239f45369f36d: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:59.343272 10213 tablet_bootstrap.cc:492] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba: Bootstrap starting.
I20260810 03:50:59.344146 10213 tablet_bootstrap.cc:654] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:59.344451 10216 tablet_bootstrap.cc:492] T ae57381d7cdc40f0b02935bcea0f0d84 P d899c3a6763243fa8bf0376062da8249: No bootstrap required, opened a new log
I20260810 03:50:59.344465 10210 tablet_bootstrap.cc:492] T ae57381d7cdc40f0b02935bcea0f0d84 P 9950a0f607014fd18fa239f45369f36d: No bootstrap required, opened a new log
I20260810 03:50:59.344530 10216 ts_tablet_manager.cc:1397] T ae57381d7cdc40f0b02935bcea0f0d84 P d899c3a6763243fa8bf0376062da8249: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:50:59.344540 10210 ts_tablet_manager.cc:1397] T ae57381d7cdc40f0b02935bcea0f0d84 P 9950a0f607014fd18fa239f45369f36d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:50:59.344969 10210 raft_consensus.cc:348] T ae57381d7cdc40f0b02935bcea0f0d84 P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.344980 10216 raft_consensus.cc:348] T ae57381d7cdc40f0b02935bcea0f0d84 P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.345088 10210 raft_consensus.cc:374] T ae57381d7cdc40f0b02935bcea0f0d84 P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:59.345089 10216 raft_consensus.cc:374] T ae57381d7cdc40f0b02935bcea0f0d84 P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:59.345121 10210 raft_consensus.cc:729] T ae57381d7cdc40f0b02935bcea0f0d84 P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9950a0f607014fd18fa239f45369f36d, State: Initialized, Role: FOLLOWER
I20260810 03:50:59.345124 10216 raft_consensus.cc:729] T ae57381d7cdc40f0b02935bcea0f0d84 P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d899c3a6763243fa8bf0376062da8249, State: Initialized, Role: FOLLOWER
I20260810 03:50:59.345237 10210 consensus_queue.cc:260] T ae57381d7cdc40f0b02935bcea0f0d84 P 9950a0f607014fd18fa239f45369f36d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.345255 10216 consensus_queue.cc:260] T ae57381d7cdc40f0b02935bcea0f0d84 P d899c3a6763243fa8bf0376062da8249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.345513 10210 ts_tablet_manager.cc:1428] T ae57381d7cdc40f0b02935bcea0f0d84 P 9950a0f607014fd18fa239f45369f36d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:50:59.345510 10216 ts_tablet_manager.cc:1428] T ae57381d7cdc40f0b02935bcea0f0d84 P d899c3a6763243fa8bf0376062da8249: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:50:59.345760 10220 tablet_replica.cc:439] T ae57381d7cdc40f0b02935bcea0f0d84 P 9950a0f607014fd18fa239f45369f36d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } } }
I20260810 03:50:59.345734 10222 tablet_replica.cc:439] T ae57381d7cdc40f0b02935bcea0f0d84 P d899c3a6763243fa8bf0376062da8249: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } } }
I20260810 03:50:59.345914 10220 tablet_replica.cc:442] T ae57381d7cdc40f0b02935bcea0f0d84 P 9950a0f607014fd18fa239f45369f36d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:50:59.345933 10222 tablet_replica.cc:442] T ae57381d7cdc40f0b02935bcea0f0d84 P d899c3a6763243fa8bf0376062da8249: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:50:59.347040 10213 tablet_bootstrap.cc:492] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba: No bootstrap required, opened a new log
I20260810 03:50:59.347132 10213 ts_tablet_manager.cc:1397] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 03:50:59.347599 10213 raft_consensus.cc:348] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.347728 10213 raft_consensus.cc:374] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:59.347769 10213 raft_consensus.cc:729] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3ee14cc82a4ad397efd5443ee068ba, State: Initialized, Role: FOLLOWER
I20260810 03:50:59.347903 10213 consensus_queue.cc:260] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.348235 10213 ts_tablet_manager.cc:1428] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:50:59.348515 10236 tablet_replica.cc:439] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } } }
I20260810 03:50:59.348654 10236 tablet_replica.cc:442] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba: This TxnStatusTablet replica's current role is: FOLLOWER
W20260810 03:50:59.438297 10157 tablet.cc:2369] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:50:59.451730 9911 tablet.cc:2369] T ae57381d7cdc40f0b02935bcea0f0d84 P d899c3a6763243fa8bf0376062da8249: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:50:59.565569 10236 raft_consensus.cc:482] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:50:59.565737 10236 raft_consensus.cc:504] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.566246 10236 leader_election.cc:288] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9950a0f607014fd18fa239f45369f36d (127.27.153.65:45569), d899c3a6763243fa8bf0376062da8249 (127.27.153.66:40937)
I20260810 03:50:59.566805 9597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae57381d7cdc40f0b02935bcea0f0d84" candidate_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9950a0f607014fd18fa239f45369f36d" is_pre_election: true
I20260810 03:50:59.566828 9827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae57381d7cdc40f0b02935bcea0f0d84" candidate_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d899c3a6763243fa8bf0376062da8249" is_pre_election: true
I20260810 03:50:59.566951 9597 raft_consensus.cc:2454] T ae57381d7cdc40f0b02935bcea0f0d84 P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3ee14cc82a4ad397efd5443ee068ba in term 0.
I20260810 03:50:59.566995 9827 raft_consensus.cc:2454] T ae57381d7cdc40f0b02935bcea0f0d84 P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a3ee14cc82a4ad397efd5443ee068ba in term 0.
I20260810 03:50:59.567273 9954 leader_election.cc:302] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9950a0f607014fd18fa239f45369f36d, 9a3ee14cc82a4ad397efd5443ee068ba; no voters:
I20260810 03:50:59.567507 10236 raft_consensus.cc:2781] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:50:59.567576 10236 raft_consensus.cc:482] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:50:59.567611 10236 raft_consensus.cc:3037] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:59.568632 10236 raft_consensus.cc:504] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.568996 10236 leader_election.cc:288] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba [CANDIDATE]: Term 1 election: Requested vote from peers 9950a0f607014fd18fa239f45369f36d (127.27.153.65:45569), d899c3a6763243fa8bf0376062da8249 (127.27.153.66:40937)
I20260810 03:50:59.569399 9597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae57381d7cdc40f0b02935bcea0f0d84" candidate_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9950a0f607014fd18fa239f45369f36d"
I20260810 03:50:59.569442 9827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae57381d7cdc40f0b02935bcea0f0d84" candidate_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d899c3a6763243fa8bf0376062da8249"
I20260810 03:50:59.569535 9597 raft_consensus.cc:3037] T ae57381d7cdc40f0b02935bcea0f0d84 P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:59.569556 9827 raft_consensus.cc:3037] T ae57381d7cdc40f0b02935bcea0f0d84 P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:59.570484 9597 raft_consensus.cc:2454] T ae57381d7cdc40f0b02935bcea0f0d84 P 9950a0f607014fd18fa239f45369f36d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3ee14cc82a4ad397efd5443ee068ba in term 1.
I20260810 03:50:59.570494 9827 raft_consensus.cc:2454] T ae57381d7cdc40f0b02935bcea0f0d84 P d899c3a6763243fa8bf0376062da8249 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a3ee14cc82a4ad397efd5443ee068ba in term 1.
I20260810 03:50:59.570787 9956 leader_election.cc:302] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a3ee14cc82a4ad397efd5443ee068ba, d899c3a6763243fa8bf0376062da8249; no voters:
I20260810 03:50:59.570958 10236 raft_consensus.cc:2781] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:59.571089 10236 raft_consensus.cc:686] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba [term 1 LEADER]: Becoming Leader. State: Replica: 9a3ee14cc82a4ad397efd5443ee068ba, State: Running, Role: LEADER
I20260810 03:50:59.571234 10236 consensus_queue.cc:237] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.571955 10225 tablet_replica.cc:439] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba: TxnStatusTablet state changed. Reason: New leader 9a3ee14cc82a4ad397efd5443ee068ba. Latest consensus state: current_term: 1 leader_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } } }
I20260810 03:50:59.572068 10225 tablet_replica.cc:442] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:50:59.572314 10284 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 03:50:59.572460 10284 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 03:50:59.573086 9296 catalog_manager.cc:5495] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a3ee14cc82a4ad397efd5443ee068ba (127.27.153.67). New cstate: current_term: 1 leader_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } health_report { overall_health: HEALTHY } } }
I20260810 03:50:59.591698 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.588301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52462 (local address 127.27.153.67:33099)
0810 03:50:59.588453 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:59.588458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:59.588472 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:50:59.588614 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:59.588617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:59.588665 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:59.588724 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:50:59.588976 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.589486 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.590348 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.590566 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.590664 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.590966 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:50:59.591051 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:50:59.591118 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:59.591550 (+ 432us) server_negotiation.cc:299] Negotiation successful
0810 03:50:59.591607 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 03:50:59.591444 10277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.588200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:33099 (local address 127.0.0.1:52462)
0810 03:50:59.588343 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:59.588360 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:59.588479 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:59.588756 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:59.588759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:59.588767 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:50:59.588873 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.588878 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.589588 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.589591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.590215 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.590223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.590677 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.590681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.590772 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.590825 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:50:59.591127 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:50:59.591147 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 03:50:59.591210 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 03:50:59.591315 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 03:50:59.594506 9827 raft_consensus.cc:1261] T ae57381d7cdc40f0b02935bcea0f0d84 P d899c3a6763243fa8bf0376062da8249 [term 1 FOLLOWER]: Refusing update from remote peer 9a3ee14cc82a4ad397efd5443ee068ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:59.594971 9597 raft_consensus.cc:1261] T ae57381d7cdc40f0b02935bcea0f0d84 P 9950a0f607014fd18fa239f45369f36d [term 1 FOLLOWER]: Refusing update from remote peer 9a3ee14cc82a4ad397efd5443ee068ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:59.594970 10236 consensus_queue.cc:1035] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:59.595388 10225 consensus_queue.cc:1035] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:59.596959 10222 tablet_replica.cc:439] T ae57381d7cdc40f0b02935bcea0f0d84 P d899c3a6763243fa8bf0376062da8249: TxnStatusTablet state changed. Reason: New leader 9a3ee14cc82a4ad397efd5443ee068ba. Latest consensus state: current_term: 1 leader_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } } }
I20260810 03:50:59.597082 10222 tablet_replica.cc:442] T ae57381d7cdc40f0b02935bcea0f0d84 P d899c3a6763243fa8bf0376062da8249: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:50:59.597396 10220 tablet_replica.cc:439] T ae57381d7cdc40f0b02935bcea0f0d84 P 9950a0f607014fd18fa239f45369f36d: TxnStatusTablet state changed. Reason: New leader 9a3ee14cc82a4ad397efd5443ee068ba. Latest consensus state: current_term: 1 leader_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } } }
I20260810 03:50:59.597492 10220 tablet_replica.cc:442] T ae57381d7cdc40f0b02935bcea0f0d84 P 9950a0f607014fd18fa239f45369f36d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:50:59.598722 10236 tablet_replica.cc:439] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } } }
I20260810 03:50:59.598831 10236 tablet_replica.cc:442] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:50:59.599282 10222 tablet_replica.cc:439] T ae57381d7cdc40f0b02935bcea0f0d84 P d899c3a6763243fa8bf0376062da8249: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } } }
I20260810 03:50:59.599397 10222 tablet_replica.cc:442] T ae57381d7cdc40f0b02935bcea0f0d84 P d899c3a6763243fa8bf0376062da8249: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:50:59.600808 10230 mvcc.cc:205] Tried to move back new op lower bound from 7316823488894828544 to 7316823488805318656. Current Snapshot: MvccSnapshot[applied={T|T < 7316823488894828544 or (T in {7316823488894828544})}]
I20260810 03:50:59.602099 10219 tablet_replica.cc:439] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } } }
I20260810 03:50:59.602211 10219 tablet_replica.cc:442] T ae57381d7cdc40f0b02935bcea0f0d84 P 9a3ee14cc82a4ad397efd5443ee068ba: This TxnStatusTablet replica's current role is: LEADER
I20260810 03:50:59.604121 10220 tablet_replica.cc:439] T ae57381d7cdc40f0b02935bcea0f0d84 P 9950a0f607014fd18fa239f45369f36d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } } }
I20260810 03:50:59.604241 10220 tablet_replica.cc:442] T ae57381d7cdc40f0b02935bcea0f0d84 P 9950a0f607014fd18fa239f45369f36d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 03:50:59.609361 10297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.603626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36100 (local address 127.27.153.66:40937)
0810 03:50:59.604651 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:59.604660 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:59.604673 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:50:59.604769 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:59.604772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:59.604991 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:59.605464 (+ 473us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:50:59.605564 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.606371 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.607444 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.607883 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.607997 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.608282 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:50:59.608908 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:50:59.609029 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:59.609126 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 03:50:59.609197 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":941,"thread_start_us":136,"threads_started":1}
I20260810 03:50:59.621547 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.617696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:39447 (local address 127.27.153.67:33099)
0810 03:50:59.617835 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:59.617840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:59.617995 (+ 155us) server_negotiation.cc:407] Connection header received
0810 03:50:59.618131 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:59.618133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:59.618183 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:59.618264 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:50:59.618559 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.619084 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.620411 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.620651 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.620751 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.621111 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:50:59.621194 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:50:59.621262 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:59.621364 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 03:50:59.621419 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:50:59.621915 10300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.617553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:33099 (local address 127.27.153.66:39447)
0810 03:50:59.617912 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:59.617927 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:59.618045 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:59.618267 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:59.618270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:59.618277 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:50:59.618412 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.618419 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.619190 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.619193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.620268 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.620282 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.620774 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.620777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.620886 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.620948 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:50:59.621279 (+ 331us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:50:59.621286 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:50:59.621344 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 03:50:59.621435 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":122,"threads_started":1}
I20260810 03:50:59.629837 9597 raft_consensus.cc:1261] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9950a0f607014fd18fa239f45369f36d [term 1 FOLLOWER]: Refusing update from remote peer d899c3a6763243fa8bf0376062da8249: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:59.629922 10075 raft_consensus.cc:1261] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9a3ee14cc82a4ad397efd5443ee068ba [term 1 FOLLOWER]: Refusing update from remote peer d899c3a6763243fa8bf0376062da8249: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:50:59.630430 10222 consensus_queue.cc:1035] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:59.630679 10269 consensus_queue.cc:1035] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:50:59.656761 10312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.652436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:40937 (local address 127.27.153.67:51015)
0810 03:50:59.653005 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:59.653021 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:59.653178 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:59.653612 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:59.653615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:59.653624 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:50:59.653781 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.653789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.654674 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.654678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.655340 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.655351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.655860 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.655864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.655967 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.656049 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:50:59.656405 (+ 356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:50:59.656414 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 03:50:59.656485 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 03:50:59.656578 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":101,"threads_started":1}
I20260810 03:50:59.656991 10297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.652621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:51015 (local address 127.27.153.66:40937)
0810 03:50:59.652782 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:59.652787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:59.653089 (+ 302us) server_negotiation.cc:407] Connection header received
0810 03:50:59.653314 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:59.653317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:59.653376 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:59.653459 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:50:59.653892 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.654565 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.655494 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.655754 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.655848 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.656232 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:50:59.656319 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:50:59.656391 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:59.656785 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 03:50:59.656853 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:50:59.685472 9294 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:59.685678 9294 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:50:59.688158 9294 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94: Sending DeleteTablet for 3 replicas of tablet f5f4d2dbda604158b7e4a3bc8eccbe59
I20260810 03:50:59.689123 9558 tablet_service.cc:1514] Processing DeleteTablet for tablet f5f4d2dbda604158b7e4a3bc8eccbe59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:59 UTC) from {username='oryx', principal='oryx/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:52052
I20260810 03:50:59.689213 9798 tablet_service.cc:1514] Processing DeleteTablet for tablet f5f4d2dbda604158b7e4a3bc8eccbe59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:59 UTC) from {username='oryx', principal='oryx/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:36092
I20260810 03:50:59.689342 10038 tablet_service.cc:1514] Processing DeleteTablet for tablet f5f4d2dbda604158b7e4a3bc8eccbe59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:50:59 UTC) from {username='oryx', principal='oryx/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:52442
I20260810 03:50:59.689401 10257 tablet_replica.cc:330] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249: stopping tablet replica
I20260810 03:50:59.689517 10257 raft_consensus.cc:2229] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:50:59.689541 10252 tablet_replica.cc:330] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9950a0f607014fd18fa239f45369f36d: stopping tablet replica
I20260810 03:50:59.689663 10252 raft_consensus.cc:2229] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9950a0f607014fd18fa239f45369f36d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:59.689781 10252 raft_consensus.cc:2258] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9950a0f607014fd18fa239f45369f36d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:59.689850 10257 raft_consensus.cc:2258] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:59.689936 10256 tablet_replica.cc:330] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9a3ee14cc82a4ad397efd5443ee068ba: stopping tablet replica
I20260810 03:50:59.690040 10256 raft_consensus.cc:2229] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9a3ee14cc82a4ad397efd5443ee068ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:50:59.690135 10256 raft_consensus.cc:2258] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9a3ee14cc82a4ad397efd5443ee068ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:50:59.690625 10257 ts_tablet_manager.cc:1905] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:59.693783 10257 ts_tablet_manager.cc:1918] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:50:59.693872 10257 log.cc:1196] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/f5f4d2dbda604158b7e4a3bc8eccbe59
I20260810 03:50:59.694113 10257 ts_tablet_manager.cc:1939] T f5f4d2dbda604158b7e4a3bc8eccbe59 P d899c3a6763243fa8bf0376062da8249: Deleting consensus metadata
I20260810 03:50:59.694924 9278 catalog_manager.cc:4841] TS d899c3a6763243fa8bf0376062da8249 (127.27.153.66:40937): tablet f5f4d2dbda604158b7e4a3bc8eccbe59 (table test-table [id=07502a0bf34344b19bcde02d2c18a6ae]) successfully deleted
I20260810 03:50:59.696486 10252 ts_tablet_manager.cc:1905] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9950a0f607014fd18fa239f45369f36d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:59.697835 10256 ts_tablet_manager.cc:1905] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9a3ee14cc82a4ad397efd5443ee068ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:50:59.702060 10252 ts_tablet_manager.cc:1918] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9950a0f607014fd18fa239f45369f36d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:50:59.702155 10252 log.cc:1196] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9950a0f607014fd18fa239f45369f36d: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-0/wal/wals/f5f4d2dbda604158b7e4a3bc8eccbe59
I20260810 03:50:59.702419 10252 ts_tablet_manager.cc:1939] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9950a0f607014fd18fa239f45369f36d: Deleting consensus metadata
I20260810 03:50:59.703277 9275 catalog_manager.cc:4841] TS 9950a0f607014fd18fa239f45369f36d (127.27.153.65:45569): tablet f5f4d2dbda604158b7e4a3bc8eccbe59 (table test-table [id=07502a0bf34344b19bcde02d2c18a6ae]) successfully deleted
I20260810 03:50:59.705374 10256 ts_tablet_manager.cc:1918] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9a3ee14cc82a4ad397efd5443ee068ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:50:59.705469 10256 log.cc:1196] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9a3ee14cc82a4ad397efd5443ee068ba: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/f5f4d2dbda604158b7e4a3bc8eccbe59
I20260810 03:50:59.705745 10256 ts_tablet_manager.cc:1939] T f5f4d2dbda604158b7e4a3bc8eccbe59 P 9a3ee14cc82a4ad397efd5443ee068ba: Deleting consensus metadata
I20260810 03:50:59.706640 9277 catalog_manager.cc:4841] TS 9a3ee14cc82a4ad397efd5443ee068ba (127.27.153.67:33099): tablet f5f4d2dbda604158b7e4a3bc8eccbe59 (table test-table [id=07502a0bf34344b19bcde02d2c18a6ae]) successfully deleted
Aug 10 03:50:59 dist-test-slave-pww0 krb5kdc[9222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 03:50:59 dist-test-slave-pww0 krb5kdc[9222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:50:59 dist-test-slave-pww0 krb5kdc[9222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 03:50:59.737527 9387 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:50:59.728705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39428 (local address 127.27.153.126:37893)
0810 03:50:59.728844 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:59.728849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:59.729168 (+ 319us) server_negotiation.cc:407] Connection header received
0810 03:50:59.729417 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:59.729421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:59.729481 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:59.729540 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:59.730721 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.731592 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.732856 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.733239 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.733329 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.737384 (+ 4055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39428: BlockingRecv error: recv got EOF from 127.0.0.1:39428 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 03:50:59 dist-test-slave-pww0 krb5kdc[9222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM
I20260810 03:50:59.749469 9387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.739711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39442 (local address 127.27.153.126:37893)
0810 03:50:59.739835 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:59.739839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:59.740348 (+ 509us) server_negotiation.cc:407] Connection header received
0810 03:50:59.740456 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:59.740459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:59.740509 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:59.740582 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:50:59.741454 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.742105 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.743275 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.743566 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.743665 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.746407 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:50:59.746410 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:50:59.746412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:50:59.746454 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:50:59.747854 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:59.748280 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:59.748283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:59.748284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:59.748365 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:50:59.748619 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:50:59.748623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:50:59.748625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:50:59.748820 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:50:59.748940 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:59.749184 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 03:50:59.749310 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 03:50:59.752537 9294 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:50:59.753043 9294 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:50:59.758193 9798 tablet_service.cc:1467] Processing CreateTablet for tablet a2ae5fc0639847c1b2f0aa50ee5c125a (DEFAULT_TABLE table=test-table [id=fdc074ee1a844d8d943b039ebef5cce2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:59.758193 9558 tablet_service.cc:1467] Processing CreateTablet for tablet a2ae5fc0639847c1b2f0aa50ee5c125a (DEFAULT_TABLE table=test-table [id=fdc074ee1a844d8d943b039ebef5cce2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:59.758536 9798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2ae5fc0639847c1b2f0aa50ee5c125a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:59.758538 9558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2ae5fc0639847c1b2f0aa50ee5c125a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:59.758467 10038 tablet_service.cc:1467] Processing CreateTablet for tablet a2ae5fc0639847c1b2f0aa50ee5c125a (DEFAULT_TABLE table=test-table [id=fdc074ee1a844d8d943b039ebef5cce2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:50:59.758754 10038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2ae5fc0639847c1b2f0aa50ee5c125a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:50:59.761091 10210 tablet_bootstrap.cc:492] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d: Bootstrap starting.
I20260810 03:50:59.761387 10216 tablet_bootstrap.cc:492] T a2ae5fc0639847c1b2f0aa50ee5c125a P d899c3a6763243fa8bf0376062da8249: Bootstrap starting.
I20260810 03:50:59.762082 10210 tablet_bootstrap.cc:654] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:59.762341 10216 tablet_bootstrap.cc:654] T a2ae5fc0639847c1b2f0aa50ee5c125a P d899c3a6763243fa8bf0376062da8249: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:59.763334 10210 tablet_bootstrap.cc:492] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d: No bootstrap required, opened a new log
I20260810 03:50:59.763406 10213 tablet_bootstrap.cc:492] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9a3ee14cc82a4ad397efd5443ee068ba: Bootstrap starting.
I20260810 03:50:59.763437 10210 ts_tablet_manager.cc:1397] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 03:50:59.764122 10210 raft_consensus.cc:348] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.764243 10210 raft_consensus.cc:374] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:59.764281 10210 raft_consensus.cc:729] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9950a0f607014fd18fa239f45369f36d, State: Initialized, Role: FOLLOWER
I20260810 03:50:59.764364 10213 tablet_bootstrap.cc:654] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9a3ee14cc82a4ad397efd5443ee068ba: Neither blocks nor log segments found. Creating new log.
I20260810 03:50:59.764426 10210 consensus_queue.cc:260] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.764699 10210 ts_tablet_manager.cc:1428] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:50:59.765758 10213 tablet_bootstrap.cc:492] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9a3ee14cc82a4ad397efd5443ee068ba: No bootstrap required, opened a new log
I20260810 03:50:59.765863 10213 ts_tablet_manager.cc:1397] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9a3ee14cc82a4ad397efd5443ee068ba: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:50:59.766263 10213 raft_consensus.cc:348] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.766357 10213 raft_consensus.cc:374] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:59.766377 10213 raft_consensus.cc:729] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3ee14cc82a4ad397efd5443ee068ba, State: Initialized, Role: FOLLOWER
I20260810 03:50:59.766491 10213 consensus_queue.cc:260] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9a3ee14cc82a4ad397efd5443ee068ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.766714 10213 ts_tablet_manager.cc:1428] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9a3ee14cc82a4ad397efd5443ee068ba: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 03:50:59.767127 10216 tablet_bootstrap.cc:492] T a2ae5fc0639847c1b2f0aa50ee5c125a P d899c3a6763243fa8bf0376062da8249: No bootstrap required, opened a new log
I20260810 03:50:59.767227 10216 ts_tablet_manager.cc:1397] T a2ae5fc0639847c1b2f0aa50ee5c125a P d899c3a6763243fa8bf0376062da8249: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260810 03:50:59.767787 10216 raft_consensus.cc:348] T a2ae5fc0639847c1b2f0aa50ee5c125a P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.767916 10216 raft_consensus.cc:374] T a2ae5fc0639847c1b2f0aa50ee5c125a P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:50:59.767947 10216 raft_consensus.cc:729] T a2ae5fc0639847c1b2f0aa50ee5c125a P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d899c3a6763243fa8bf0376062da8249, State: Initialized, Role: FOLLOWER
I20260810 03:50:59.768075 10216 consensus_queue.cc:260] T a2ae5fc0639847c1b2f0aa50ee5c125a P d899c3a6763243fa8bf0376062da8249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.768376 10216 ts_tablet_manager.cc:1428] T a2ae5fc0639847c1b2f0aa50ee5c125a P d899c3a6763243fa8bf0376062da8249: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:50:59.982184 10220 raft_consensus.cc:482] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:50:59.982350 10220 raft_consensus.cc:504] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.983676 10220 leader_election.cc:288] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d899c3a6763243fa8bf0376062da8249 (127.27.153.66:40937), 9a3ee14cc82a4ad397efd5443ee068ba (127.27.153.67:33099)
I20260810 03:50:59.989140 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.984113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:55259 (local address 127.27.153.67:33099)
0810 03:50:59.984378 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:59.984383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:59.984499 (+ 116us) server_negotiation.cc:407] Connection header received
0810 03:50:59.984677 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:59.984680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:59.984768 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:59.984854 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:59.985199 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.985993 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.987769 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.988411 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.988548 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.988605 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:59.988914 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 03:50:59.988996 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171}
I20260810 03:50:59.989140 10297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.984221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:33869 (local address 127.27.153.66:40937)
0810 03:50:59.984393 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 03:50:59.984397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:50:59.984597 (+ 200us) server_negotiation.cc:407] Connection header received
0810 03:50:59.984718 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:50:59.984721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:50:59.984766 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:50:59.984854 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:50:59.985199 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.985993 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.987769 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.988417 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.988551 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.988605 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 03:50:59.988915 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 03:50:59.988996 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 03:50:59.989216 10344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.983972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:33099 (local address 127.27.153.65:55259)
0810 03:50:59.984359 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:59.984375 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:59.984517 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:59.984881 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:59.984884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:59.984932 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:59.985087 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.985095 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.986143 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.986145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.987592 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.987607 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.988573 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.988578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.988753 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.988799 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 03:50:59.988887 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 03:50:59.989005 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":159,"spinlock_wait_cycles":10112,"thread_start_us":93,"threads_started":1}
I20260810 03:50:59.989216 10345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:50:59.984003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:40937 (local address 127.27.153.65:33869)
0810 03:50:59.984516 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0810 03:50:59.984528 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:50:59.984606 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:50:59.984881 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:50:59.984885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:50:59.984931 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:50:59.985097 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.985102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.986138 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.986142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.987592 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:50:59.987607 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:50:59.988573 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:50:59.988578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:50:59.988774 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:50:59.988803 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 03:50:59.988883 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 03:50:59.989005 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":189,"spinlock_wait_cycles":32640,"thread_start_us":107,"threads_started":1}
I20260810 03:50:59.989816 10075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ae5fc0639847c1b2f0aa50ee5c125a" candidate_uuid: "9950a0f607014fd18fa239f45369f36d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" is_pre_election: true
I20260810 03:50:59.989993 10075 raft_consensus.cc:2454] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9950a0f607014fd18fa239f45369f36d in term 0.
I20260810 03:50:59.990039 9827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ae5fc0639847c1b2f0aa50ee5c125a" candidate_uuid: "9950a0f607014fd18fa239f45369f36d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d899c3a6763243fa8bf0376062da8249" is_pre_election: true
I20260810 03:50:59.990196 9827 raft_consensus.cc:2454] T a2ae5fc0639847c1b2f0aa50ee5c125a P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9950a0f607014fd18fa239f45369f36d in term 0.
I20260810 03:50:59.990430 9471 leader_election.cc:302] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9950a0f607014fd18fa239f45369f36d, 9a3ee14cc82a4ad397efd5443ee068ba; no voters:
I20260810 03:50:59.990767 10220 raft_consensus.cc:2781] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:50:59.990828 10220 raft_consensus.cc:482] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:50:59.990854 10220 raft_consensus.cc:3037] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:59.991994 10220 raft_consensus.cc:504] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.992384 10220 leader_election.cc:288] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d [CANDIDATE]: Term 1 election: Requested vote from peers d899c3a6763243fa8bf0376062da8249 (127.27.153.66:40937), 9a3ee14cc82a4ad397efd5443ee068ba (127.27.153.67:33099)
I20260810 03:50:59.992751 9827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ae5fc0639847c1b2f0aa50ee5c125a" candidate_uuid: "9950a0f607014fd18fa239f45369f36d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d899c3a6763243fa8bf0376062da8249"
I20260810 03:50:59.992755 10075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ae5fc0639847c1b2f0aa50ee5c125a" candidate_uuid: "9950a0f607014fd18fa239f45369f36d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3ee14cc82a4ad397efd5443ee068ba"
I20260810 03:50:59.992884 10075 raft_consensus.cc:3037] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:59.992884 9827 raft_consensus.cc:3037] T a2ae5fc0639847c1b2f0aa50ee5c125a P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:50:59.994047 10075 raft_consensus.cc:2454] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9a3ee14cc82a4ad397efd5443ee068ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9950a0f607014fd18fa239f45369f36d in term 1.
I20260810 03:50:59.994047 9827 raft_consensus.cc:2454] T a2ae5fc0639847c1b2f0aa50ee5c125a P d899c3a6763243fa8bf0376062da8249 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9950a0f607014fd18fa239f45369f36d in term 1.
I20260810 03:50:59.994405 9471 leader_election.cc:302] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9950a0f607014fd18fa239f45369f36d, 9a3ee14cc82a4ad397efd5443ee068ba; no voters:
I20260810 03:50:59.994589 10220 raft_consensus.cc:2781] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:50:59.994768 10220 raft_consensus.cc:686] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d [term 1 LEADER]: Becoming Leader. State: Replica: 9950a0f607014fd18fa239f45369f36d, State: Running, Role: LEADER
I20260810 03:50:59.995015 10220 consensus_queue.cc:237] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:50:59.997536 9297 catalog_manager.cc:5495] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9950a0f607014fd18fa239f45369f36d (127.27.153.65). New cstate: current_term: 1 leader_uuid: "9950a0f607014fd18fa239f45369f36d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } health_report { overall_health: UNKNOWN } } }
I20260810 03:51:00.014959 10356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:00.011071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52062 (local address 127.27.153.65:45569)
0810 03:51:00.011393 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:00.011400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:00.011411 (+ 11us) server_negotiation.cc:407] Connection header received
0810 03:51:00.011535 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:00.011538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:00.011602 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:00.011718 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:51:00.011990 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:00.012540 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:00.013395 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:00.013650 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:00.013740 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:00.014021 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:51:00.014513 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:51:00.014620 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:00.014709 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 03:51:00.014773 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":114,"threads_started":1}
I20260810 03:51:00.017941 9827 raft_consensus.cc:1261] T a2ae5fc0639847c1b2f0aa50ee5c125a P d899c3a6763243fa8bf0376062da8249 [term 1 FOLLOWER]: Refusing update from remote peer 9950a0f607014fd18fa239f45369f36d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:51:00.017941 10075 raft_consensus.cc:1261] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9a3ee14cc82a4ad397efd5443ee068ba [term 1 FOLLOWER]: Refusing update from remote peer 9950a0f607014fd18fa239f45369f36d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:51:00.018549 10220 consensus_queue.cc:1035] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:51:00.018783 10346 consensus_queue.cc:1035] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:51:00.031028 9297 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:51:00.031230 9297 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:51:00.033582 9297 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94: Sending DeleteTablet for 3 replicas of tablet a2ae5fc0639847c1b2f0aa50ee5c125a
I20260810 03:51:00.034341 9558 tablet_service.cc:1514] Processing DeleteTablet for tablet a2ae5fc0639847c1b2f0aa50ee5c125a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:51:00 UTC) from {username='oryx', principal='oryx/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:52052
I20260810 03:51:00.034489 9798 tablet_service.cc:1514] Processing DeleteTablet for tablet a2ae5fc0639847c1b2f0aa50ee5c125a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:51:00 UTC) from {username='oryx', principal='oryx/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:36092
I20260810 03:51:00.034551 10252 tablet_replica.cc:330] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d: stopping tablet replica
I20260810 03:51:00.034663 10252 raft_consensus.cc:2229] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:51:00.035065 10252 raft_consensus.cc:2258] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:51:00.035264 9297 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 03:51:00.035503 10257 tablet_replica.cc:330] T a2ae5fc0639847c1b2f0aa50ee5c125a P d899c3a6763243fa8bf0376062da8249: stopping tablet replica
I20260810 03:51:00.035665 10257 raft_consensus.cc:2229] T a2ae5fc0639847c1b2f0aa50ee5c125a P d899c3a6763243fa8bf0376062da8249 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:51:00.035825 10257 raft_consensus.cc:2258] T a2ae5fc0639847c1b2f0aa50ee5c125a P d899c3a6763243fa8bf0376062da8249 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 03:51:00.035806 9297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:51:00.036744 10038 tablet_service.cc:1514] Processing DeleteTablet for tablet a2ae5fc0639847c1b2f0aa50ee5c125a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:51:00 UTC) from {username='oryx', principal='oryx/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:52442
I20260810 03:51:00.036875 10257 ts_tablet_manager.cc:1905] T a2ae5fc0639847c1b2f0aa50ee5c125a P d899c3a6763243fa8bf0376062da8249: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:51:00.036937 10256 tablet_replica.cc:330] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9a3ee14cc82a4ad397efd5443ee068ba: stopping tablet replica
I20260810 03:51:00.037043 10256 raft_consensus.cc:2229] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9a3ee14cc82a4ad397efd5443ee068ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:51:00.037169 10256 raft_consensus.cc:2258] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9a3ee14cc82a4ad397efd5443ee068ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:51:00.038115 10252 ts_tablet_manager.cc:1905] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:51:00.039597 10257 ts_tablet_manager.cc:1918] T a2ae5fc0639847c1b2f0aa50ee5c125a P d899c3a6763243fa8bf0376062da8249: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:51:00.039676 10257 log.cc:1196] T a2ae5fc0639847c1b2f0aa50ee5c125a P d899c3a6763243fa8bf0376062da8249: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/a2ae5fc0639847c1b2f0aa50ee5c125a
I20260810 03:51:00.039916 10257 ts_tablet_manager.cc:1939] T a2ae5fc0639847c1b2f0aa50ee5c125a P d899c3a6763243fa8bf0376062da8249: Deleting consensus metadata
I20260810 03:51:00.040792 10252 ts_tablet_manager.cc:1918] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:51:00.040889 10252 log.cc:1196] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-0/wal/wals/a2ae5fc0639847c1b2f0aa50ee5c125a
I20260810 03:51:00.040908 9278 catalog_manager.cc:4841] TS d899c3a6763243fa8bf0376062da8249 (127.27.153.66:40937): tablet a2ae5fc0639847c1b2f0aa50ee5c125a (table test-table [id=fdc074ee1a844d8d943b039ebef5cce2]) successfully deleted
I20260810 03:51:00.041136 10252 ts_tablet_manager.cc:1939] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9950a0f607014fd18fa239f45369f36d: Deleting consensus metadata
I20260810 03:51:00.041960 9558 tablet_service.cc:1467] Processing CreateTablet for tablet 9f8ded56f9714b97b33e6f561d010ddc (DEFAULT_TABLE table=test-table [id=5e8b7344023c42e3bbb097aa20924523]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:51:00.042114 9798 tablet_service.cc:1467] Processing CreateTablet for tablet 9f8ded56f9714b97b33e6f561d010ddc (DEFAULT_TABLE table=test-table [id=5e8b7344023c42e3bbb097aa20924523]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:51:00.042233 9558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f8ded56f9714b97b33e6f561d010ddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:51:00.042352 9798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f8ded56f9714b97b33e6f561d010ddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:51:00.042531 10038 tablet_service.cc:1467] Processing CreateTablet for tablet 9f8ded56f9714b97b33e6f561d010ddc (DEFAULT_TABLE table=test-table [id=5e8b7344023c42e3bbb097aa20924523]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:51:00.042941 10038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f8ded56f9714b97b33e6f561d010ddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:51:00.044667 10216 tablet_bootstrap.cc:492] T 9f8ded56f9714b97b33e6f561d010ddc P d899c3a6763243fa8bf0376062da8249: Bootstrap starting.
I20260810 03:51:00.044920 10256 ts_tablet_manager.cc:1905] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9a3ee14cc82a4ad397efd5443ee068ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:51:00.045518 10216 tablet_bootstrap.cc:654] T 9f8ded56f9714b97b33e6f561d010ddc P d899c3a6763243fa8bf0376062da8249: Neither blocks nor log segments found. Creating new log.
I20260810 03:51:00.045961 10213 tablet_bootstrap.cc:492] T 9f8ded56f9714b97b33e6f561d010ddc P 9a3ee14cc82a4ad397efd5443ee068ba: Bootstrap starting.
I20260810 03:51:00.046310 9275 catalog_manager.cc:4841] TS 9950a0f607014fd18fa239f45369f36d (127.27.153.65:45569): tablet a2ae5fc0639847c1b2f0aa50ee5c125a (table test-table [id=fdc074ee1a844d8d943b039ebef5cce2]) successfully deleted
I20260810 03:51:00.046802 10213 tablet_bootstrap.cc:654] T 9f8ded56f9714b97b33e6f561d010ddc P 9a3ee14cc82a4ad397efd5443ee068ba: Neither blocks nor log segments found. Creating new log.
I20260810 03:51:00.047155 10256 ts_tablet_manager.cc:1918] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9a3ee14cc82a4ad397efd5443ee068ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:51:00.047225 10256 log.cc:1196] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9a3ee14cc82a4ad397efd5443ee068ba: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/a2ae5fc0639847c1b2f0aa50ee5c125a
I20260810 03:51:00.047497 10256 ts_tablet_manager.cc:1939] T a2ae5fc0639847c1b2f0aa50ee5c125a P 9a3ee14cc82a4ad397efd5443ee068ba: Deleting consensus metadata
I20260810 03:51:00.047785 10210 tablet_bootstrap.cc:492] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d: Bootstrap starting.
I20260810 03:51:00.048082 10213 tablet_bootstrap.cc:492] T 9f8ded56f9714b97b33e6f561d010ddc P 9a3ee14cc82a4ad397efd5443ee068ba: No bootstrap required, opened a new log
I20260810 03:51:00.048168 10213 ts_tablet_manager.cc:1397] T 9f8ded56f9714b97b33e6f561d010ddc P 9a3ee14cc82a4ad397efd5443ee068ba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 03:51:00.048589 10213 raft_consensus.cc:348] T 9f8ded56f9714b97b33e6f561d010ddc P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:51:00.048718 10213 raft_consensus.cc:374] T 9f8ded56f9714b97b33e6f561d010ddc P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:51:00.048751 10210 tablet_bootstrap.cc:654] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d: Neither blocks nor log segments found. Creating new log.
I20260810 03:51:00.048761 10213 raft_consensus.cc:729] T 9f8ded56f9714b97b33e6f561d010ddc P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3ee14cc82a4ad397efd5443ee068ba, State: Initialized, Role: FOLLOWER
I20260810 03:51:00.048877 10213 consensus_queue.cc:260] T 9f8ded56f9714b97b33e6f561d010ddc P 9a3ee14cc82a4ad397efd5443ee068ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:51:00.049141 10213 ts_tablet_manager.cc:1428] T 9f8ded56f9714b97b33e6f561d010ddc P 9a3ee14cc82a4ad397efd5443ee068ba: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:51:00.049456 9277 catalog_manager.cc:4841] TS 9a3ee14cc82a4ad397efd5443ee068ba (127.27.153.67:33099): tablet a2ae5fc0639847c1b2f0aa50ee5c125a (table test-table [id=fdc074ee1a844d8d943b039ebef5cce2]) successfully deleted
I20260810 03:51:00.050024 10210 tablet_bootstrap.cc:492] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d: No bootstrap required, opened a new log
I20260810 03:51:00.050117 10210 ts_tablet_manager.cc:1397] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 03:51:00.050675 10210 raft_consensus.cc:348] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:51:00.050814 10210 raft_consensus.cc:374] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:51:00.050861 10210 raft_consensus.cc:729] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9950a0f607014fd18fa239f45369f36d, State: Initialized, Role: FOLLOWER
I20260810 03:51:00.050989 10210 consensus_queue.cc:260] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:51:00.051301 10210 ts_tablet_manager.cc:1428] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:51:00.053174 10216 tablet_bootstrap.cc:492] T 9f8ded56f9714b97b33e6f561d010ddc P d899c3a6763243fa8bf0376062da8249: No bootstrap required, opened a new log
I20260810 03:51:00.053295 10216 ts_tablet_manager.cc:1397] T 9f8ded56f9714b97b33e6f561d010ddc P d899c3a6763243fa8bf0376062da8249: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260810 03:51:00.053704 10216 raft_consensus.cc:348] T 9f8ded56f9714b97b33e6f561d010ddc P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:51:00.053815 10216 raft_consensus.cc:374] T 9f8ded56f9714b97b33e6f561d010ddc P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:51:00.053838 10216 raft_consensus.cc:729] T 9f8ded56f9714b97b33e6f561d010ddc P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d899c3a6763243fa8bf0376062da8249, State: Initialized, Role: FOLLOWER
I20260810 03:51:00.053932 10216 consensus_queue.cc:260] T 9f8ded56f9714b97b33e6f561d010ddc P d899c3a6763243fa8bf0376062da8249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:51:00.054154 10216 ts_tablet_manager.cc:1428] T 9f8ded56f9714b97b33e6f561d010ddc P d899c3a6763243fa8bf0376062da8249: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 03:51:00.240864 10363 raft_consensus.cc:482] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:51:00.241024 10363 raft_consensus.cc:504] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:51:00.241528 10363 leader_election.cc:288] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d899c3a6763243fa8bf0376062da8249 (127.27.153.66:40937), 9a3ee14cc82a4ad397efd5443ee068ba (127.27.153.67:33099)
I20260810 03:51:00.242007 9827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8ded56f9714b97b33e6f561d010ddc" candidate_uuid: "9950a0f607014fd18fa239f45369f36d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d899c3a6763243fa8bf0376062da8249" is_pre_election: true
I20260810 03:51:00.242019 10075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8ded56f9714b97b33e6f561d010ddc" candidate_uuid: "9950a0f607014fd18fa239f45369f36d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" is_pre_election: true
I20260810 03:51:00.242157 9827 raft_consensus.cc:2454] T 9f8ded56f9714b97b33e6f561d010ddc P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9950a0f607014fd18fa239f45369f36d in term 0.
I20260810 03:51:00.242197 10075 raft_consensus.cc:2454] T 9f8ded56f9714b97b33e6f561d010ddc P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9950a0f607014fd18fa239f45369f36d in term 0.
I20260810 03:51:00.242545 9471 leader_election.cc:302] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9950a0f607014fd18fa239f45369f36d, 9a3ee14cc82a4ad397efd5443ee068ba; no voters:
I20260810 03:51:00.242738 10363 raft_consensus.cc:2781] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:51:00.242812 10363 raft_consensus.cc:482] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:51:00.242846 10363 raft_consensus.cc:3037] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:51:00.244006 10363 raft_consensus.cc:504] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:51:00.244375 10363 leader_election.cc:288] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d [CANDIDATE]: Term 1 election: Requested vote from peers d899c3a6763243fa8bf0376062da8249 (127.27.153.66:40937), 9a3ee14cc82a4ad397efd5443ee068ba (127.27.153.67:33099)
I20260810 03:51:00.244807 9827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8ded56f9714b97b33e6f561d010ddc" candidate_uuid: "9950a0f607014fd18fa239f45369f36d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d899c3a6763243fa8bf0376062da8249"
I20260810 03:51:00.244807 10075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8ded56f9714b97b33e6f561d010ddc" candidate_uuid: "9950a0f607014fd18fa239f45369f36d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a3ee14cc82a4ad397efd5443ee068ba"
I20260810 03:51:00.244951 10075 raft_consensus.cc:3037] T 9f8ded56f9714b97b33e6f561d010ddc P 9a3ee14cc82a4ad397efd5443ee068ba [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:51:00.244951 9827 raft_consensus.cc:3037] T 9f8ded56f9714b97b33e6f561d010ddc P d899c3a6763243fa8bf0376062da8249 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:51:00.246130 9827 raft_consensus.cc:2454] T 9f8ded56f9714b97b33e6f561d010ddc P d899c3a6763243fa8bf0376062da8249 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9950a0f607014fd18fa239f45369f36d in term 1.
I20260810 03:51:00.246130 10075 raft_consensus.cc:2454] T 9f8ded56f9714b97b33e6f561d010ddc P 9a3ee14cc82a4ad397efd5443ee068ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9950a0f607014fd18fa239f45369f36d in term 1.
I20260810 03:51:00.246467 9471 leader_election.cc:302] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9950a0f607014fd18fa239f45369f36d, 9a3ee14cc82a4ad397efd5443ee068ba; no voters:
I20260810 03:51:00.246631 10363 raft_consensus.cc:2781] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:51:00.246721 10363 raft_consensus.cc:686] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d [term 1 LEADER]: Becoming Leader. State: Replica: 9950a0f607014fd18fa239f45369f36d, State: Running, Role: LEADER
I20260810 03:51:00.246851 10363 consensus_queue.cc:237] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } }
I20260810 03:51:00.248605 9297 catalog_manager.cc:5495] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9950a0f607014fd18fa239f45369f36d (127.27.153.65). New cstate: current_term: 1 leader_uuid: "9950a0f607014fd18fa239f45369f36d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9950a0f607014fd18fa239f45369f36d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 45569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 } health_report { overall_health: UNKNOWN } } }
I20260810 03:51:00.303332 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:00.299906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:39153 (local address 127.27.153.67:33099)
0810 03:51:00.300051 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:00.300055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:00.300167 (+ 112us) server_negotiation.cc:407] Connection header received
0810 03:51:00.300340 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:00.300343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:00.300400 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:00.300474 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:51:00.300711 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:00.301290 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:00.302217 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:00.302456 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:00.302552 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:00.302855 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:51:00.302941 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:51:00.303001 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:00.303113 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 03:51:00.303174 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 03:51:00.303333 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:00.299739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:33099 (local address 127.27.153.65:39153)
0810 03:51:00.300049 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:00.300062 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:00.300196 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:00.300496 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:00.300498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:00.300505 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:51:00.300615 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:00.300620 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:00.301432 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:00.301436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:00.302092 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:00.302103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:00.302576 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:00.302580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:00.302664 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:00.302720 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:51:00.303022 (+ 302us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:51:00.303028 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 03:51:00.303096 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 03:51:00.303199 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":96,"threads_started":1}
I20260810 03:51:00.309896 10075 raft_consensus.cc:1261] T 9f8ded56f9714b97b33e6f561d010ddc P 9a3ee14cc82a4ad397efd5443ee068ba [term 1 FOLLOWER]: Refusing update from remote peer 9950a0f607014fd18fa239f45369f36d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:51:00.310278 9827 raft_consensus.cc:1261] T 9f8ded56f9714b97b33e6f561d010ddc P d899c3a6763243fa8bf0376062da8249 [term 1 FOLLOWER]: Refusing update from remote peer 9950a0f607014fd18fa239f45369f36d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:51:00.310308 10346 consensus_queue.cc:1035] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a3ee14cc82a4ad397efd5443ee068ba" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:51:00.310689 10346 consensus_queue.cc:1035] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d899c3a6763243fa8bf0376062da8249" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:51:00.328153 10356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:00.324746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:40141 (local address 127.27.153.65:45569)
0810 03:51:00.324867 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:00.324872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:00.325100 (+ 228us) server_negotiation.cc:407] Connection header received
0810 03:51:00.325211 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:00.325214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:00.325275 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:00.325349 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:51:00.325665 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:00.326278 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:00.327106 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:00.327322 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:00.327393 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:00.327699 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:51:00.327792 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:51:00.327845 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:00.327964 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 03:51:00.328025 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 03:51:00.328154 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:00.324697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:45569 (local address 127.27.153.67:40141)
0810 03:51:00.325014 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:00.325026 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:00.325128 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:00.325399 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:00.325402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:00.325411 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 03:51:00.325552 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:00.325559 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:00.326380 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:00.326383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:00.327010 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:00.327019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:00.327425 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:00.327429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:00.327539 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:00.327577 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 03:51:00.327870 (+ 293us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 03:51:00.327878 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 03:51:00.327942 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 03:51:00.328022 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":127,"threads_started":1}
I20260810 03:51:00.349696 9294 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:51:00.349869 9294 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:51:00.352087 9294 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 648f6a718a104afa8530c6ffc0acbc94: Sending DeleteTablet for 3 replicas of tablet 9f8ded56f9714b97b33e6f561d010ddc
I20260810 03:51:00.352684 9558 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f8ded56f9714b97b33e6f561d010ddc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:51:00 UTC) from {username='oryx', principal='oryx/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:52052
I20260810 03:51:00.352763 10038 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f8ded56f9714b97b33e6f561d010ddc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:51:00 UTC) from {username='oryx', principal='oryx/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:52442
I20260810 03:51:00.352931 9798 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f8ded56f9714b97b33e6f561d010ddc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:51:00 UTC) from {username='oryx', principal='oryx/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:36092
I20260810 03:51:00.353142 10252 tablet_replica.cc:330] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d: stopping tablet replica
I20260810 03:51:00.353142 10257 tablet_replica.cc:330] T 9f8ded56f9714b97b33e6f561d010ddc P d899c3a6763243fa8bf0376062da8249: stopping tablet replica
I20260810 03:51:00.353271 10257 raft_consensus.cc:2229] T 9f8ded56f9714b97b33e6f561d010ddc P d899c3a6763243fa8bf0376062da8249 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:51:00.353281 10252 raft_consensus.cc:2229] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:51:00.353315 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 9430
I20260810 03:51:00.353396 10257 raft_consensus.cc:2258] T 9f8ded56f9714b97b33e6f561d010ddc P d899c3a6763243fa8bf0376062da8249 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:51:00.353487 10252 raft_consensus.cc:2258] T 9f8ded56f9714b97b33e6f561d010ddc P 9950a0f607014fd18fa239f45369f36d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:51:00.353657 10256 tablet_replica.cc:330] T 9f8ded56f9714b97b33e6f561d010ddc P 9a3ee14cc82a4ad397efd5443ee068ba: stopping tablet replica
I20260810 03:51:00.353765 10256 raft_consensus.cc:2229] T 9f8ded56f9714b97b33e6f561d010ddc P 9a3ee14cc82a4ad397efd5443ee068ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:51:00.353873 10256 raft_consensus.cc:2258] T 9f8ded56f9714b97b33e6f561d010ddc P 9a3ee14cc82a4ad397efd5443ee068ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:51:00.368742 10257 ts_tablet_manager.cc:1905] T 9f8ded56f9714b97b33e6f561d010ddc P d899c3a6763243fa8bf0376062da8249: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:51:00.372052 10257 ts_tablet_manager.cc:1918] T 9f8ded56f9714b97b33e6f561d010ddc P d899c3a6763243fa8bf0376062da8249: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:51:00.372162 10257 log.cc:1196] T 9f8ded56f9714b97b33e6f561d010ddc P d899c3a6763243fa8bf0376062da8249: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/9f8ded56f9714b97b33e6f561d010ddc
I20260810 03:51:00.372396 10257 ts_tablet_manager.cc:1939] T 9f8ded56f9714b97b33e6f561d010ddc P d899c3a6763243fa8bf0376062da8249: Deleting consensus metadata
I20260810 03:51:00.372411 10256 ts_tablet_manager.cc:1905] T 9f8ded56f9714b97b33e6f561d010ddc P 9a3ee14cc82a4ad397efd5443ee068ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:51:00.373368 9278 catalog_manager.cc:4841] TS d899c3a6763243fa8bf0376062da8249 (127.27.153.66:40937): tablet 9f8ded56f9714b97b33e6f561d010ddc (table test-table [id=5e8b7344023c42e3bbb097aa20924523]) successfully deleted
I20260810 03:51:00.374717 10256 ts_tablet_manager.cc:1918] T 9f8ded56f9714b97b33e6f561d010ddc P 9a3ee14cc82a4ad397efd5443ee068ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 03:51:00.374801 10256 log.cc:1196] T 9f8ded56f9714b97b33e6f561d010ddc P 9a3ee14cc82a4ad397efd5443ee068ba: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/9f8ded56f9714b97b33e6f561d010ddc
I20260810 03:51:00.375023 10256 ts_tablet_manager.cc:1939] T 9f8ded56f9714b97b33e6f561d010ddc P 9a3ee14cc82a4ad397efd5443ee068ba: Deleting consensus metadata
I20260810 03:51:00.375865 9277 catalog_manager.cc:4841] TS 9a3ee14cc82a4ad397efd5443ee068ba (127.27.153.67:33099): tablet 9f8ded56f9714b97b33e6f561d010ddc (table test-table [id=5e8b7344023c42e3bbb097aa20924523]) successfully deleted
W20260810 03:51:00.381105 9275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.153.65:45569 (error 108)
I20260810 03:51:00.381175 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 9683
I20260810 03:51:00.389300 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:00.387189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:45569 (local address 127.0.0.1:52070)
0810 03:51:00.389110 (+ 1921us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:00.389194 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:45569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1848,"thread_start_us":1652,"threads_started":1}
W20260810 03:51:00.389562 9275 catalog_manager.cc:4584] TS 9950a0f607014fd18fa239f45369f36d (127.27.153.65:45569): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9f8ded56f9714b97b33e6f561d010ddc: Network error: Client connection negotiation failed: client connection to 127.27.153.65:45569: connect: Connection refused (error 111)
I20260810 03:51:00.399569 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 9924
I20260810 03:51:00.413668 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 9241
2026-08-10T03:51:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3802 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:51:00 dist-test-slave-pww0 krb5kdc[10399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:51:00 dist-test-slave-pww0 krb5kdc[10399](info): set up 2 sockets
Aug 10 03:51:00 dist-test-slave-pww0 krb5kdc[10399](info): commencing operation
krb5kdc: starting...
W20260810 03:51:01.625557 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.179s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:51:01 dist-test-slave-pww0 krb5kdc[10399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:51:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:51:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:51:01.759256 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:35319
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:33255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:35319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:51:01.875681 10424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:51:01.876000 10424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:51:01.876060 10424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:51:01.880255 10424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:51:01.880340 10424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:51:01.880367 10424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:51:01.880389 10424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:51:01.880409 10424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:51:01.886193 10424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:33255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:35319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:35319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:51:01.887399 10424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:51:01.888525 10424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:51:01.898437 10434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:01.898505 10437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:01.898473 10432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:01.898460 10424 server_base.cc:1034] running on GCE node
I20260810 03:51:01.899322 10424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:51:01.900537 10424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:51:01.901736 10424 hybrid_clock.cc:648] HybridClock initialized: now 1786333861901715 us; error 37 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T03:51:01Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1494 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:51:01 dist-test-slave-pww0 krb5kdc[10446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:51:01 dist-test-slave-pww0 krb5kdc[10446](info): set up 2 sockets
Aug 10 03:51:01 dist-test-slave-pww0 krb5kdc[10446](info): commencing operation
krb5kdc: starting...
W20260810 03:51:03.133876 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:51:03 dist-test-slave-pww0 krb5kdc[10446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:51:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:51:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:51:03.266582 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:39581
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:39581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:51:03.382212 10469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:51:03.382529 10469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:51:03.382604 10469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:51:03.386801 10469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:51:03.386883 10469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:51:03.386911 10469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:51:03.386933 10469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:51:03.386955 10469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:51:03.392652 10469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:39581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:39581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:51:03.393813 10469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:51:03.394868 10469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:51:03.403702 10489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:03.403702 10486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:03.403848 10492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:03.404101 10469 server_base.cc:1034] running on GCE node
I20260810 03:51:03.404693 10469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:51:03.405850 10469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:51:03.407011 10469 hybrid_clock.cc:648] HybridClock initialized: now 1786333863406987 us; error 36 us; skew 500 ppm
Aug 10 03:51:03 dist-test-slave-pww0 krb5kdc[10446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333863, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:51:03.409797 10469 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:51:03.410728 10469 webserver.cc:460] Webserver started at http://127.27.153.126:35511/ using document root <none> and password file <none>
I20260810 03:51:03.411253 10469 fs_manager.cc:359] Metadata directory not provided
I20260810 03:51:03.411324 10469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:51:03.411545 10469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:51:03.413671 10469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "1bfc37b82ac34c4da8f40dbbadba7822"
format_stamp: "Formatted at 2026-08-10 03:51:03 on dist-test-slave-pww0"
server_key: "1eb36bab2217248bb7ca9a6ba7f138f4"
server_key_iv: "52dbc77a8cad734901cc556749e01ea6"
server_key_version: "encryptionkey@0"
I20260810 03:51:03.414233 10469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "1bfc37b82ac34c4da8f40dbbadba7822"
format_stamp: "Formatted at 2026-08-10 03:51:03 on dist-test-slave-pww0"
server_key: "1eb36bab2217248bb7ca9a6ba7f138f4"
server_key_iv: "52dbc77a8cad734901cc556749e01ea6"
server_key_version: "encryptionkey@0"
I20260810 03:51:03.418393 10469 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:51:03.420928 10504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:03.422210 10469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 03:51:03.422371 10469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "1bfc37b82ac34c4da8f40dbbadba7822"
format_stamp: "Formatted at 2026-08-10 03:51:03 on dist-test-slave-pww0"
server_key: "1eb36bab2217248bb7ca9a6ba7f138f4"
server_key_iv: "52dbc77a8cad734901cc556749e01ea6"
server_key_version: "encryptionkey@0"
I20260810 03:51:03.422480 10469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:51:03.435154 10469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:51:03.438220 10469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:51:03.438427 10469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:51:03.445645 10469 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:39581
I20260810 03:51:03.445657 10584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:39581 every 8 connection(s)
I20260810 03:51:03.446736 10469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:51:03.449749 10587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:51:03.454480 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 10469
I20260810 03:51:03.454628 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:51:03.454895 28261 external_mini_cluster.cc:1428] Setting key 34994181083d0ea19de0b0418ddb12de
I20260810 03:51:03.455837 10587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822: Bootstrap starting.
I20260810 03:51:03.458794 10587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822: Neither blocks nor log segments found. Creating new log.
I20260810 03:51:03.459776 10587 log.cc:824] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822: Log is configured to *not* fsync() on all Append() calls
I20260810 03:51:03.462054 10587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822: No bootstrap required, opened a new log
Aug 10 03:51:03 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333863, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:51:03.466425 10587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfc37b82ac34c4da8f40dbbadba7822" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39581 } }
I20260810 03:51:03.466744 10587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:51:03.466805 10587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bfc37b82ac34c4da8f40dbbadba7822, State: Initialized, Role: FOLLOWER
I20260810 03:51:03.467327 10587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfc37b82ac34c4da8f40dbbadba7822" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39581 } }
I20260810 03:51:03.467499 10587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:51:03.467612 10587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:51:03.467726 10587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:51:03.468776 10587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfc37b82ac34c4da8f40dbbadba7822" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39581 } }
I20260810 03:51:03.469180 10587 leader_election.cc:302] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bfc37b82ac34c4da8f40dbbadba7822; no voters:
I20260810 03:51:03.469542 10587 leader_election.cc:288] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:51:03.469664 10595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:51:03.469888 10595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [term 1 LEADER]: Becoming Leader. State: Replica: 1bfc37b82ac34c4da8f40dbbadba7822, State: Running, Role: LEADER
I20260810 03:51:03.470310 10595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfc37b82ac34c4da8f40dbbadba7822" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39581 } }
I20260810 03:51:03.470609 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:03.456349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40400 (local address 127.27.153.126:39581)
0810 03:51:03.456813 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:03.456834 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:03.456878 (+ 44us) server_negotiation.cc:407] Connection header received
0810 03:51:03.457810 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:03.457837 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:03.458295 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:03.458711 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:03.459505 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:03.460815 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:03.461752 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:03.462449 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:03.462613 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:03.465122 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:51:03.465128 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:51:03.465149 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:51:03.465185 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:51:03.467085 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:51:03.467604 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:51:03.467611 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:51:03.467616 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:51:03.467765 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:51:03.468110 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:51:03.468120 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:51:03.468122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:51:03.468573 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:51:03.468749 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:03.469145 (+ 396us) server_negotiation.cc:299] Negotiation successful
0810 03:51:03.469482 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":117,"threads_started":1}
I20260810 03:51:03.470836 10587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:51:03.472016 10596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bfc37b82ac34c4da8f40dbbadba7822. Latest consensus state: current_term: 1 leader_uuid: "1bfc37b82ac34c4da8f40dbbadba7822" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfc37b82ac34c4da8f40dbbadba7822" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39581 } } }
I20260810 03:51:03.472016 10598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bfc37b82ac34c4da8f40dbbadba7822" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfc37b82ac34c4da8f40dbbadba7822" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39581 } } }
I20260810 03:51:03.472254 10598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [sys.catalog]: This master's current role is: LEADER
I20260810 03:51:03.472200 10596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [sys.catalog]: This master's current role is: LEADER
I20260810 03:51:03.473193 10613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:51:03.476194 10613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:51:03.482537 10613 catalog_manager.cc:1347] Generated new cluster ID: 00f54e6a400d47439a7f102d7e881faa
I20260810 03:51:03.482626 10613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:51:03.490845 10613 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:51:03.491958 10613 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:51:03.502676 10613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822: Generated new TSK 0
I20260810 03:51:03.503685 10613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.153.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
I20260810 03:51:03.538710 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:39581
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:51:03.655050 10629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:51:03.655370 10629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:51:03.655436 10629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:51:03.659631 10629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:51:03.659713 10629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:51:03.659780 10629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:51:03.665380 10629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:39581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:51:03.666522 10629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:51:03.667604 10629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:51:03.677939 10640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:03.678082 10629 server_base.cc:1034] running on GCE node
W20260810 03:51:03.677927 10649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:03.677932 10645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:03.678615 10629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:51:03.679356 10629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:51:03.680541 10629 hybrid_clock.cc:648] HybridClock initialized: now 1786333863680522 us; error 33 us; skew 500 ppm
Aug 10 03:51:03 dist-test-slave-pww0 krb5kdc[10446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333863, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:51:03.683487 10629 init.cc:377] Logged in from keytab as kudu/127.27.153.65@KRBTEST.COM (short username kudu)
I20260810 03:51:03.684471 10629 webserver.cc:460] Webserver started at http://127.27.153.65:41651/ using document root <none> and password file <none>
I20260810 03:51:03.685053 10629 fs_manager.cc:359] Metadata directory not provided
I20260810 03:51:03.685124 10629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:51:03.685328 10629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:51:03.687729 10629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "ea6f12605a6b4a769add0c8691207b3a"
format_stamp: "Formatted at 2026-08-10 03:51:03 on dist-test-slave-pww0"
server_key: "851f17179877b7c4be4fd5b5aa52a3a1"
server_key_iv: "4b96958f19ab2f012200e7268b9c8ce3"
server_key_version: "encryptionkey@0"
I20260810 03:51:03.688308 10629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "ea6f12605a6b4a769add0c8691207b3a"
format_stamp: "Formatted at 2026-08-10 03:51:03 on dist-test-slave-pww0"
server_key: "851f17179877b7c4be4fd5b5aa52a3a1"
server_key_iv: "4b96958f19ab2f012200e7268b9c8ce3"
server_key_version: "encryptionkey@0"
I20260810 03:51:03.692237 10629 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:51:03.694764 10662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:03.696099 10629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:51:03.696224 10629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "ea6f12605a6b4a769add0c8691207b3a"
format_stamp: "Formatted at 2026-08-10 03:51:03 on dist-test-slave-pww0"
server_key: "851f17179877b7c4be4fd5b5aa52a3a1"
server_key_iv: "4b96958f19ab2f012200e7268b9c8ce3"
server_key_version: "encryptionkey@0"
I20260810 03:51:03.696331 10629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:51:03.725279 10629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:51:03.728693 10629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:51:03.728892 10629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:51:03.729504 10629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:51:03.730381 10629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:51:03.730428 10629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:03.730482 10629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:51:03.730512 10629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:03.739408 10629 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:39247
I20260810 03:51:03.739419 10864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:39247 every 8 connection(s)
I20260810 03:51:03.740399 10629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:51:03.747198 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 10629
I20260810 03:51:03.747350 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:51:03.747705 28261 external_mini_cluster.cc:1428] Setting key af353d3db25d9dee9465ff9f8078898b
Aug 10 03:51:03 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333863, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:51:03.754184 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:03.742301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:52629 (local address 127.27.153.126:39581)
0810 03:51:03.742440 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:03.742444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:03.743510 (+ 1066us) server_negotiation.cc:407] Connection header received
0810 03:51:03.744418 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:03.744421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:03.744481 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:03.744555 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:03.745749 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:03.746292 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:03.747735 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:03.748003 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:03.748098 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:03.750729 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:51:03.750732 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:51:03.750734 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:51:03.750760 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:51:03.752128 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:51:03.752638 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:51:03.752641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:51:03.752643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:51:03.752732 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:51:03.753058 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:51:03.753061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:51:03.753063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:51:03.753259 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:51:03.753348 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:03.753876 (+ 528us) server_negotiation.cc:299] Negotiation successful
0810 03:51:03.754012 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 03:51:03.755064 10870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:03.742693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.65:52629)
0810 03:51:03.743337 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:03.743381 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:03.744209 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:03.744748 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:03.744756 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:03.745114 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:03.745595 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:03.745608 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:03.746420 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:03.746423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:03.747599 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:03.747611 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:03.748287 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:03.748290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:03.748447 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:03.749047 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:03.749064 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:03.750543 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:51:03.752246 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:51:03.752252 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:51:03.752270 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:51:03.752536 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:51:03.752817 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:51:03.752821 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:51:03.752823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:51:03.752970 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:51:03.753362 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:51:03.753369 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:51:03.753634 (+ 265us) client_negotiation.cc:769] Sending connection context
0810 03:51:03.753816 (+ 182us) client_negotiation.cc:240] Negotiation successful
0810 03:51:03.754183 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":107,"threads_started":1}
I20260810 03:51:03.756410 10865 heartbeater.cc:347] Connected to a master server at 127.27.153.126:39581
I20260810 03:51:03.756655 10865 heartbeater.cc:464] Registering TS with master...
I20260810 03:51:03.757187 10865 heartbeater.cc:511] Master 127.27.153.126:39581 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.153.66@KRBTEST.COM; defaulting to no policy
I20260810 03:51:03.758780 10533 ts_manager.cc:194] Registered new tserver with Master: ea6f12605a6b4a769add0c8691207b3a (127.27.153.65:39247)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.66@KRBTEST.COM" created.
I20260810 03:51:03.760123 10533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:52629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
I20260810 03:51:03.778465 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:39581
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:51:03.897991 10879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:51:03.898298 10879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:51:03.898343 10879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:51:03.903728 10879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:51:03.903808 10879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:51:03.903874 10879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:51:03.909595 10879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:39581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:51:03.910912 10879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:51:03.912179 10879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:51:03.922410 10884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:03.922506 10879 server_base.cc:1034] running on GCE node
W20260810 03:51:03.922413 10885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:03.922410 10889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:03.923164 10879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:51:03.923902 10879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:51:03.925062 10879 hybrid_clock.cc:648] HybridClock initialized: now 1786333863925042 us; error 33 us; skew 500 ppm
Aug 10 03:51:03 dist-test-slave-pww0 krb5kdc[10446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333863, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:51:03.928082 10879 init.cc:377] Logged in from keytab as kudu/127.27.153.66@KRBTEST.COM (short username kudu)
I20260810 03:51:03.929059 10879 webserver.cc:460] Webserver started at http://127.27.153.66:34999/ using document root <none> and password file <none>
I20260810 03:51:03.929626 10879 fs_manager.cc:359] Metadata directory not provided
I20260810 03:51:03.929698 10879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:51:03.929894 10879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:51:03.932200 10879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "1e8bbaf17ed2429983058d3d76b48c49"
format_stamp: "Formatted at 2026-08-10 03:51:03 on dist-test-slave-pww0"
server_key: "33b429473179d56683130cddfd27fcfc"
server_key_iv: "519e3e2c9fe6e903567ae9f1c3c73c7e"
server_key_version: "encryptionkey@0"
I20260810 03:51:03.932794 10879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "1e8bbaf17ed2429983058d3d76b48c49"
format_stamp: "Formatted at 2026-08-10 03:51:03 on dist-test-slave-pww0"
server_key: "33b429473179d56683130cddfd27fcfc"
server_key_iv: "519e3e2c9fe6e903567ae9f1c3c73c7e"
server_key_version: "encryptionkey@0"
I20260810 03:51:03.936822 10879 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 03:51:03.939522 10900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:03.940960 10879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:51:03.941087 10879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "1e8bbaf17ed2429983058d3d76b48c49"
format_stamp: "Formatted at 2026-08-10 03:51:03 on dist-test-slave-pww0"
server_key: "33b429473179d56683130cddfd27fcfc"
server_key_iv: "519e3e2c9fe6e903567ae9f1c3c73c7e"
server_key_version: "encryptionkey@0"
I20260810 03:51:03.941195 10879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:51:03.962674 10879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:51:03.966096 10879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:51:03.966305 10879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:51:03.966943 10879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:51:03.967866 10879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:51:03.967919 10879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:03.967976 10879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:51:03.968003 10879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:03.977324 10879 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:41301
I20260810 03:51:03.977362 11099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:41301 every 8 connection(s)
I20260810 03:51:03.978271 10879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:51:03.978591 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 10879
I20260810 03:51:03.978719 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:51:03.979018 28261 external_mini_cluster.cc:1428] Setting key 199e036d1b53ff4ca93926f7d70dd6d6
Aug 10 03:51:03 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333863, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.153.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.67@KRBTEST.COM" created.
I20260810 03:51:03.992969 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:03.980197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:59491 (local address 127.27.153.126:39581)
0810 03:51:03.980336 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:03.980341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:03.981553 (+ 1212us) server_negotiation.cc:407] Connection header received
0810 03:51:03.982449 (+ 896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:03.982453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:03.982516 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:03.982600 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:03.983949 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:03.984671 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:03.986053 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:03.986411 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:03.986533 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:03.989180 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:51:03.989185 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:51:03.989188 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:51:03.989227 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:51:03.990645 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:51:03.991316 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:51:03.991319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:51:03.991321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:51:03.991400 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:51:03.991777 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:51:03.991779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:51:03.991780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:51:03.991977 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:51:03.992052 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:03.992662 (+ 610us) server_negotiation.cc:299] Negotiation successful
0810 03:51:03.992800 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 03:51:03.993817 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:03.980646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.66:59491)
0810 03:51:03.981378 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:03.981434 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:03.982234 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:03.982812 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:03.982820 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:03.983200 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:03.983784 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:03.983801 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:03.984836 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:03.984839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:03.985893 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:03.985908 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:03.986576 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:03.986579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:03.986750 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:03.987406 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:03.987423 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:03.988955 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:51:03.990775 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:51:03.990781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:51:03.990795 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:51:03.991222 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:51:03.991508 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:51:03.991511 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:51:03.991512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:51:03.991694 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:51:03.992089 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:51:03.992095 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:51:03.992387 (+ 292us) client_negotiation.cc:769] Sending connection context
0810 03:51:03.992602 (+ 215us) client_negotiation.cc:240] Negotiation successful
0810 03:51:03.992925 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":117,"threads_started":1}
I20260810 03:51:03.995160 11100 heartbeater.cc:347] Connected to a master server at 127.27.153.126:39581
I20260810 03:51:03.995494 11100 heartbeater.cc:464] Registering TS with master...
I20260810 03:51:03.996187 11100 heartbeater.cc:511] Master 127.27.153.126:39581 requested a full tablet report, sending...
I20260810 03:51:03.997376 10533 ts_manager.cc:194] Registered new tserver with Master: 1e8bbaf17ed2429983058d3d76b48c49 (127.27.153.66:41301)
I20260810 03:51:03.997985 10533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:59491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
I20260810 03:51:04.010789 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:39581
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:51:04.127187 11114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:51:04.127552 11114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:51:04.127629 11114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:51:04.132129 11114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:51:04.132223 11114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:51:04.132311 11114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:51:04.138361 11114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:39581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:51:04.139569 11114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:51:04.140659 11114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:51:04.150823 11125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:04.150831 11121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:04.150869 11122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:04.151139 11114 server_base.cc:1034] running on GCE node
I20260810 03:51:04.151715 11114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:51:04.152426 11114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:51:04.153640 11114 hybrid_clock.cc:648] HybridClock initialized: now 1786333864153592 us; error 42 us; skew 500 ppm
Aug 10 03:51:04 dist-test-slave-pww0 krb5kdc[10446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333864, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:51:04.156773 11114 init.cc:377] Logged in from keytab as kudu/127.27.153.67@KRBTEST.COM (short username kudu)
I20260810 03:51:04.157821 11114 webserver.cc:460] Webserver started at http://127.27.153.67:41205/ using document root <none> and password file <none>
I20260810 03:51:04.158382 11114 fs_manager.cc:359] Metadata directory not provided
I20260810 03:51:04.158450 11114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:51:04.158690 11114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:51:04.160893 11114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "46a193a0e32145b19013766796ce3816"
format_stamp: "Formatted at 2026-08-10 03:51:04 on dist-test-slave-pww0"
server_key: "a7b4d40e8ef347e0b127c7275fd4fbd9"
server_key_iv: "630ff94e6b06cd593bcfa1f1f351f5a1"
server_key_version: "encryptionkey@0"
I20260810 03:51:04.161456 11114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "46a193a0e32145b19013766796ce3816"
format_stamp: "Formatted at 2026-08-10 03:51:04 on dist-test-slave-pww0"
server_key: "a7b4d40e8ef347e0b127c7275fd4fbd9"
server_key_iv: "630ff94e6b06cd593bcfa1f1f351f5a1"
server_key_version: "encryptionkey@0"
I20260810 03:51:04.165342 11114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 03:51:04.167950 11140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:04.169231 11114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:51:04.169370 11114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "46a193a0e32145b19013766796ce3816"
format_stamp: "Formatted at 2026-08-10 03:51:04 on dist-test-slave-pww0"
server_key: "a7b4d40e8ef347e0b127c7275fd4fbd9"
server_key_iv: "630ff94e6b06cd593bcfa1f1f351f5a1"
server_key_version: "encryptionkey@0"
I20260810 03:51:04.169478 11114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:51:04.180807 11114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:51:04.184567 11114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:51:04.184787 11114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:51:04.185585 11114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:51:04.188706 11114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:51:04.188767 11114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:04.188828 11114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:51:04.188858 11114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:04.202203 11114 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:39095
I20260810 03:51:04.202245 11370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:39095 every 8 connection(s)
I20260810 03:51:04.203351 11114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:51:04.212077 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 11114
I20260810 03:51:04.212225 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:51:04.212504 28261 external_mini_cluster.cc:1428] Setting key 8d9efe24a4d96dca9b0ded0d75fed1f3
Aug 10 03:51:04 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333864, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:51:04.220815 10591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:04.205481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:52661 (local address 127.27.153.126:39581)
0810 03:51:04.205644 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:04.205649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:04.206956 (+ 1307us) server_negotiation.cc:407] Connection header received
0810 03:51:04.208237 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:04.208242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:04.208307 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:04.208406 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:04.210378 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:04.211452 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:04.212849 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:04.213194 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:04.213306 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:04.216524 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:51:04.216528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:51:04.216531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:51:04.216564 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:51:04.218219 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:51:04.218874 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:51:04.218878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:51:04.218879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:51:04.218972 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:51:04.219404 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:51:04.219406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:51:04.219407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:51:04.219694 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:51:04.219794 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:04.220489 (+ 695us) server_negotiation.cc:299] Negotiation successful
0810 03:51:04.220627 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:51:04.222030 11379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:04.205877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.67:52661)
0810 03:51:04.206731 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:04.206817 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:04.207963 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:04.208613 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:04.208624 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:04.209190 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:04.210118 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:04.210145 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:04.211701 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:04.211705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:04.212705 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:04.212716 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:04.213363 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:04.213368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:04.213590 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:04.214308 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:04.214331 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:04.216333 (+ 2002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:51:04.218338 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:51:04.218345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:51:04.218364 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:51:04.218750 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:51:04.219074 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:51:04.219077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:51:04.219080 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:51:04.219306 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:51:04.219820 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:51:04.219828 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:51:04.220205 (+ 377us) client_negotiation.cc:769] Sending connection context
0810 03:51:04.220442 (+ 237us) client_negotiation.cc:240] Negotiation successful
0810 03:51:04.220859 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":147,"threads_started":1}
I20260810 03:51:04.223371 11371 heartbeater.cc:347] Connected to a master server at 127.27.153.126:39581
I20260810 03:51:04.223697 11371 heartbeater.cc:464] Registering TS with master...
I20260810 03:51:04.224383 11371 heartbeater.cc:511] Master 127.27.153.126:39581 requested a full tablet report, sending...
I20260810 03:51:04.225749 10533 ts_manager.cc:194] Registered new tserver with Master: 46a193a0e32145b19013766796ce3816 (127.27.153.67:39095)
I20260810 03:51:04.226392 10533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:52661
I20260810 03:51:04.227610 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.153.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.153.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.153.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.65.keytab.
Entry for principal oryx/127.27.153.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.65.keytab.
WARNING: no policy specified for oryx/127.27.153.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.153.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.153.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.66.keytab.
Entry for principal oryx/127.27.153.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.66.keytab.
WARNING: no policy specified for oryx/127.27.153.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.153.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.153.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.67.keytab.
Entry for principal oryx/127.27.153.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.67.keytab.
I20260810 03:51:04.314833 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 10629
I20260810 03:51:04.346834 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 10879
I20260810 03:51:04.368211 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 11114
I20260810 03:51:04.383385 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 10469
I20260810 03:51:04.418367 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:39581
--webserver_interface=127.27.153.126
--webserver_port=35511
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:39581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:51:04.534153 11396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:51:04.534449 11396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:51:04.534492 11396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:51:04.538681 11396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:51:04.538759 11396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:51:04.538777 11396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:51:04.538793 11396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:51:04.538811 11396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:51:04.544397 11396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:39581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:39581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=35511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:51:04.545552 11396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:51:04.546669 11396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:51:04.555335 11407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:04.555473 11396 server_base.cc:1034] running on GCE node
W20260810 03:51:04.555361 11411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:04.555385 11408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:04.556280 11396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:51:04.557472 11396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:51:04.558668 11396 hybrid_clock.cc:648] HybridClock initialized: now 1786333864558647 us; error 36 us; skew 500 ppm
Aug 10 03:51:04 dist-test-slave-pww0 krb5kdc[10446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333864, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:51:04.561616 11396 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:51:04.562575 11396 webserver.cc:460] Webserver started at http://127.27.153.126:35511/ using document root <none> and password file <none>
I20260810 03:51:04.563150 11396 fs_manager.cc:359] Metadata directory not provided
I20260810 03:51:04.563234 11396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:51:04.567337 11396 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:51:04.569640 11427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:04.570940 11396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 03:51:04.571058 11396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "1bfc37b82ac34c4da8f40dbbadba7822"
format_stamp: "Formatted at 2026-08-10 03:51:03 on dist-test-slave-pww0"
server_key: "1eb36bab2217248bb7ca9a6ba7f138f4"
server_key_iv: "52dbc77a8cad734901cc556749e01ea6"
server_key_version: "encryptionkey@0"
I20260810 03:51:04.571720 11396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:51:04.586958 11396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:51:04.590678 11396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:51:04.590891 11396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:51:04.598793 11396 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:39581
I20260810 03:51:04.598871 11522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:39581 every 8 connection(s)
I20260810 03:51:04.599947 11396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:51:04.604493 11524 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:51:04.606019 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 11396
I20260810 03:51:04.606364 11524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822: Bootstrap starting.
I20260810 03:51:04.606483 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:39247
--local_ip_for_outbound_sockets=127.27.153.65
--tserver_master_addrs=127.27.153.126:39581
--webserver_port=41651
--webserver_interface=127.27.153.65
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.153.65
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:51:04.624406 11524 log.cc:824] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822: Log is configured to *not* fsync() on all Append() calls
I20260810 03:51:04.631588 11524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:51:04.631961 11524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822: Bootstrap complete.
I20260810 03:51:04.636363 11524 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfc37b82ac34c4da8f40dbbadba7822" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39581 } }
I20260810 03:51:04.636822 11524 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bfc37b82ac34c4da8f40dbbadba7822, State: Initialized, Role: FOLLOWER
I20260810 03:51:04.637353 11524 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfc37b82ac34c4da8f40dbbadba7822" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39581 } }
I20260810 03:51:04.637491 11524 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:51:04.637567 11524 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:51:04.637674 11524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:51:04.639956 11524 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfc37b82ac34c4da8f40dbbadba7822" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39581 } }
I20260810 03:51:04.640317 11524 leader_election.cc:302] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bfc37b82ac34c4da8f40dbbadba7822; no voters:
I20260810 03:51:04.640712 11524 leader_election.cc:288] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 03:51:04.640856 11536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 03:51:04.641116 11536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [term 2 LEADER]: Becoming Leader. State: Replica: 1bfc37b82ac34c4da8f40dbbadba7822, State: Running, Role: LEADER
I20260810 03:51:04.641570 11536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfc37b82ac34c4da8f40dbbadba7822" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39581 } }
I20260810 03:51:04.642009 11524 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:51:04.643139 11537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1bfc37b82ac34c4da8f40dbbadba7822" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfc37b82ac34c4da8f40dbbadba7822" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39581 } } }
I20260810 03:51:04.643325 11537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [sys.catalog]: This master's current role is: LEADER
I20260810 03:51:04.643729 11538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bfc37b82ac34c4da8f40dbbadba7822. Latest consensus state: current_term: 2 leader_uuid: "1bfc37b82ac34c4da8f40dbbadba7822" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bfc37b82ac34c4da8f40dbbadba7822" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39581 } } }
I20260810 03:51:04.643818 11538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822 [sys.catalog]: This master's current role is: LEADER
I20260810 03:51:04.644450 11549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:51:04.647985 11549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:51:04.648681 11549 catalog_manager.cc:1259] Loaded cluster ID: 00f54e6a400d47439a7f102d7e881faa
I20260810 03:51:04.648751 11549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:51:04.651170 11549 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:51:04.652034 11549 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1bfc37b82ac34c4da8f40dbbadba7822: Loaded TSK: 0
I20260810 03:51:04.653355 11549 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 03:51:04.729239 11532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:51:04.729537 11532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:51:04.729583 11532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:51:04.734026 11532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:51:04.734108 11532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:51:04.734182 11532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:51:04.740017 11532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.65.keytab
--principal=oryx/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:39247
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=41651
--tserver_master_addrs=127.27.153.126:39581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:51:04.741236 11532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:51:04.742355 11532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:51:04.753412 11573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:04.753646 11575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:04.753813 11532 server_base.cc:1034] running on GCE node
W20260810 03:51:04.754007 11577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:04.754446 11532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:51:04.755211 11532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:51:04.756397 11532 hybrid_clock.cc:648] HybridClock initialized: now 1786333864756363 us; error 47 us; skew 500 ppm
Aug 10 03:51:04 dist-test-slave-pww0 krb5kdc[10446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333864, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:51:04.759909 11532 init.cc:377] Logged in from keytab as oryx/127.27.153.65@KRBTEST.COM (short username oryx)
I20260810 03:51:04.760995 11532 webserver.cc:460] Webserver started at http://127.27.153.65:41651/ using document root <none> and password file <none>
I20260810 03:51:04.761591 11532 fs_manager.cc:359] Metadata directory not provided
I20260810 03:51:04.761682 11532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:51:04.766021 11532 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:51:04.768466 11594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:04.769881 11532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:51:04.770013 11532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "ea6f12605a6b4a769add0c8691207b3a"
format_stamp: "Formatted at 2026-08-10 03:51:03 on dist-test-slave-pww0"
server_key: "851f17179877b7c4be4fd5b5aa52a3a1"
server_key_iv: "4b96958f19ab2f012200e7268b9c8ce3"
server_key_version: "encryptionkey@0"
I20260810 03:51:04.770673 11532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:51:04.802830 11532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:51:04.806140 11532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:51:04.806346 11532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:51:04.806977 11532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:51:04.807873 11532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:51:04.807924 11532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:04.807981 11532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:51:04.808008 11532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:04.816959 11532 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:39247
I20260810 03:51:04.816992 11825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:39247 every 8 connection(s)
I20260810 03:51:04.817986 11532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:51:04.825264 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 11532
I20260810 03:51:04.825732 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:41301
--local_ip_for_outbound_sockets=127.27.153.66
--tserver_master_addrs=127.27.153.126:39581
--webserver_port=34999
--webserver_interface=127.27.153.66
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.153.66
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:51:04 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:04 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:04.831640 11831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:04.820249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.65:55529)
0810 03:51:04.820967 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:04.821010 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:04.821842 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:04.822983 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:04.822991 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:04.823418 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:04.823937 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:04.823952 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:04.825236 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:04.825239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:04.826236 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:04.826247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:04.827006 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:04.827010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:04.827219 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:04.827891 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:04.827913 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:04.830737 (+ 2824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":119,"threads_started":1}
W20260810 03:51:04.832062 11830 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:04.820144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:55529 (local address 127.27.153.126:39581)
0810 03:51:04.820739 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:04.820761 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:04.821147 (+ 386us) server_negotiation.cc:407] Connection header received
0810 03:51:04.822246 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:04.822257 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:04.822558 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:04.822874 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:04.824103 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:04.825100 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:04.826357 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:04.826842 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:04.826994 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:04.831057 (+ 4063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.65:55529: BlockingRecv error: recv got EOF from 127.27.153.65:55529 (error 108)
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":134,"threads_started":1}
W20260810 03:51:04.832227 11826 heartbeater.cc:644] Failed to heartbeat to 127.27.153.126:39581 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.153.126:39581: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Aug 10 03:51:04 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:04 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:04.838182 11831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:04.832715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.65:48461)
0810 03:51:04.832899 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:04.832914 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:04.833009 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:04.833278 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:04.833281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:04.833459 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:04.833573 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:04.833580 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:04.834343 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:04.834348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:04.834910 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:04.834918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:04.835381 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:04.835385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:04.835513 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:04.836000 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:04.836013 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:04.838016 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112}
W20260810 03:51:04.838310 11830 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:04.832829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:48461 (local address 127.27.153.126:39581)
0810 03:51:04.832958 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:04.832963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:04.832979 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:51:04.833115 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:04.833119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:04.833171 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:04.833248 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:04.833692 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:04.834227 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:04.835031 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:04.835261 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:04.835355 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:04.838194 (+ 2839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.65:48461: BlockingRecv error: recv got EOF from 127.27.153.65:48461 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 03:51:04 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:04 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:04.844156 11831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:04.838887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.65:40155)
0810 03:51:04.839120 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:04.839143 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:04.839223 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:04.839448 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:04.839450 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:04.839622 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:04.839703 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:04.839708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:04.840408 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:04.840412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:04.840988 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:04.840998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:04.841452 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:04.841458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:04.841543 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:04.841982 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:04.841994 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:04.843954 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166}
W20260810 03:51:04.844200 11830 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:04.839112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:40155 (local address 127.27.153.126:39581)
0810 03:51:04.839219 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:04.839223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:04.839237 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:51:04.839310 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:04.839313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:04.839357 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:04.839419 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:04.839803 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:04.840306 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:04.841101 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:04.841316 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:04.841399 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:04.844098 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.65:40155: BlockingRecv error: recv got EOF from 127.27.153.65:40155 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260810 03:51:04.844599 11826 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:51:04.943554 11832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:51:04.943881 11832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:51:04.943929 11832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:51:04.948004 11832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:51:04.948074 11832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:51:04.948135 11832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:51:04.953636 11832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.66.keytab
--principal=oryx/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:41301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=34999
--tserver_master_addrs=127.27.153.126:39581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:51:04.954752 11832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:51:04.955850 11832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:51:04.966115 11843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:04.966204 11832 server_base.cc:1034] running on GCE node
W20260810 03:51:04.966265 11842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:04.966136 11845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:04.966881 11832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:51:04.967684 11832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:51:04.968883 11832 hybrid_clock.cc:648] HybridClock initialized: now 1786333864968861 us; error 34 us; skew 500 ppm
Aug 10 03:51:04 dist-test-slave-pww0 krb5kdc[10446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333864, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:51:04.971971 11832 init.cc:377] Logged in from keytab as oryx/127.27.153.66@KRBTEST.COM (short username oryx)
I20260810 03:51:04.972967 11832 webserver.cc:460] Webserver started at http://127.27.153.66:34999/ using document root <none> and password file <none>
I20260810 03:51:04.973524 11832 fs_manager.cc:359] Metadata directory not provided
I20260810 03:51:04.973606 11832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:51:04.977975 11832 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:51:04.980204 11856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:04.981561 11832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:51:04.981680 11832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "1e8bbaf17ed2429983058d3d76b48c49"
format_stamp: "Formatted at 2026-08-10 03:51:03 on dist-test-slave-pww0"
server_key: "33b429473179d56683130cddfd27fcfc"
server_key_iv: "519e3e2c9fe6e903567ae9f1c3c73c7e"
server_key_version: "encryptionkey@0"
I20260810 03:51:04.982228 11832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:51:04.997054 11832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:51:05.000561 11832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:51:05.000762 11832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:51:05.001412 11832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:51:05.002319 11832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:51:05.002377 11832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:05.002426 11832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:51:05.002457 11832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:05.012136 11832 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:41301
I20260810 03:51:05.012161 12054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:41301 every 8 connection(s)
I20260810 03:51:05.013239 11832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:51:05.014133 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 11832
I20260810 03:51:05.014650 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:39095
--local_ip_for_outbound_sockets=127.27.153.67
--tserver_master_addrs=127.27.153.126:39581
--webserver_port=41205
--webserver_interface=127.27.153.67
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.153.67
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:51:05 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:05 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 03:51:05.026217 11830 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:05.015435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:52801 (local address 127.27.153.126:39581)
0810 03:51:05.015599 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:05.015603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:05.017071 (+ 1468us) server_negotiation.cc:407] Connection header received
0810 03:51:05.018257 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:05.018261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:05.018347 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:05.018443 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:05.019776 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.020604 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.021825 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.022172 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.022283 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:05.026089 (+ 3806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.66:52801: BlockingRecv error: recv got EOF from 127.27.153.66:52801 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 03:51:05.027122 12058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:05.015966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.66:52801)
0810 03:51:05.016862 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:05.016930 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:05.017986 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:05.018642 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:05.018650 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:05.019033 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:05.019596 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:05.019620 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.020760 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.020763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:05.021690 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:05.021701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.022346 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.022350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:05.022501 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:05.023147 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:05.023171 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:05.025934 (+ 2763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":155,"threads_started":1}
W20260810 03:51:05.027729 12055 heartbeater.cc:644] Failed to heartbeat to 127.27.153.126:39581 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.153.126:39581: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Aug 10 03:51:05 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:05 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:05.033715 12058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:05.028228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.66:37159)
0810 03:51:05.028404 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:05.028418 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:05.028507 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:05.028753 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:05.028756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:05.028925 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:05.029025 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:05.029031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.029773 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.029776 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:05.030373 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:05.030383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.030818 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.030821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:05.030919 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:05.031423 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:05.031435 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:05.033543 (+ 2108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109}
W20260810 03:51:05.034138 11830 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:05.028361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:37159 (local address 127.27.153.126:39581)
0810 03:51:05.028488 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:05.028492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:05.028507 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:51:05.028594 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:05.028597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:05.028640 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:05.028706 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:05.029141 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.029664 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.030496 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.030713 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.030802 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:05.033719 (+ 2917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.66:37159: BlockingRecv error: recv got EOF from 127.27.153.66:37159 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 03:51:05 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:05 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:05.040262 12058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:05.034331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.66:42071)
0810 03:51:05.034445 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:05.034458 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:05.034534 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:05.034948 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:05.034951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:05.035083 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:05.035169 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:05.035175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.036366 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.036368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:05.036913 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:05.036922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.037509 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.037512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:05.037575 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:05.038121 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:05.038133 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:05.040151 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 03:51:05.040555 11830 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:05.034593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:42071 (local address 127.27.153.126:39581)
0810 03:51:05.034714 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:05.034718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:05.034731 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:51:05.034775 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:05.034777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:05.034825 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:05.034912 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:05.035265 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.036226 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.037023 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.037386 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.037488 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:05.040332 (+ 2844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.66:42071: BlockingRecv error: recv got EOF from 127.27.153.66:42071 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 03:51:05.040594 12055 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:51:05.140791 12059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:51:05.141080 12059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:51:05.141124 12059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:51:05.145514 12059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:51:05.145594 12059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:51:05.145664 12059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:51:05.151290 12059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/krb5kdc/oryx_127.27.153.67.keytab
--principal=oryx/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:39095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=41205
--tserver_master_addrs=127.27.153.126:39581
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:51:05.152534 12059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:51:05.153628 12059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:51:05.164366 12074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:05.164386 12071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:05.164465 12059 server_base.cc:1034] running on GCE node
W20260810 03:51:05.164395 12069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:05.165064 12059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:51:05.165771 12059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:51:05.166946 12059 hybrid_clock.cc:648] HybridClock initialized: now 1786333865166920 us; error 46 us; skew 500 ppm
Aug 10 03:51:05 dist-test-slave-pww0 krb5kdc[10446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333865, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:51:05.169991 12059 init.cc:377] Logged in from keytab as oryx/127.27.153.67@KRBTEST.COM (short username oryx)
I20260810 03:51:05.171041 12059 webserver.cc:460] Webserver started at http://127.27.153.67:41205/ using document root <none> and password file <none>
I20260810 03:51:05.171698 12059 fs_manager.cc:359] Metadata directory not provided
I20260810 03:51:05.171797 12059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:51:05.176244 12059 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:51:05.178527 12088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:05.179910 12059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:51:05.180047 12059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "46a193a0e32145b19013766796ce3816"
format_stamp: "Formatted at 2026-08-10 03:51:04 on dist-test-slave-pww0"
server_key: "a7b4d40e8ef347e0b127c7275fd4fbd9"
server_key_iv: "630ff94e6b06cd593bcfa1f1f351f5a1"
server_key_version: "encryptionkey@0"
I20260810 03:51:05.180619 12059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:51:05.200915 12059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:51:05.204324 12059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:51:05.204523 12059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:51:05.205165 12059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:51:05.206032 12059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:51:05.206081 12059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:05.206137 12059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:51:05.206168 12059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:05.215332 12059 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:39095
I20260810 03:51:05.215411 12315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:39095 every 8 connection(s)
I20260810 03:51:05.216454 12059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:51:05.224071 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 12059
Aug 10 03:51:05 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:05 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 03:51:05.228978 11830 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:05.218397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:45413 (local address 127.27.153.126:39581)
0810 03:51:05.218529 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:05.218533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:05.219863 (+ 1330us) server_negotiation.cc:407] Connection header received
0810 03:51:05.220815 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:05.220819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:05.220875 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:05.220960 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:05.222334 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.222892 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.224270 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.224575 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.224677 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:05.228855 (+ 4178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.67:45413: BlockingRecv error: recv got EOF from 127.27.153.67:45413 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 03:51:05.229832 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:05.218807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.67:45413)
0810 03:51:05.219631 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:05.219703 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:05.220568 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:05.221188 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:05.221196 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:05.221581 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:05.222137 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:05.222152 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.223043 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.223046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:05.224107 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:05.224119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.224787 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.224793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:05.225037 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:05.225774 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:05.225799 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:05.228726 (+ 2927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":111,"threads_started":1}
W20260810 03:51:05.230417 12316 heartbeater.cc:644] Failed to heartbeat to 127.27.153.126:39581 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.153.126:39581: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
I20260810 03:51:05.234853 12320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:05.225235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40404 (local address 127.27.153.126:39581)
0810 03:51:05.225560 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:05.225565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:05.225583 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:51:05.225630 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:05.225633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:05.225689 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:05.225781 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:05.226751 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.227527 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.228478 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.228730 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.228830 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:05.230571 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:51:05.230578 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:51:05.230593 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:51:05.230630 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:51:05.232589 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:51:05.233068 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:51:05.233075 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:51:05.233082 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:51:05.233246 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:51:05.233499 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:51:05.233501 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:51:05.233503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:51:05.233936 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:51:05.234142 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:05.234400 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 03:51:05.234674 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":105,"threads_started":1}
Aug 10 03:51:05 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:05 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:05.236673 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:05.230888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.67:37993)
0810 03:51:05.231063 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:05.231076 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:05.231155 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:05.231675 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:05.231679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:05.231852 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:05.231986 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:05.231994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.232745 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.232748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:05.233469 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:05.233481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.233927 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.233931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:05.234028 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:05.234515 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:05.234529 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:05.236520 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109}
W20260810 03:51:05.236820 11830 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:05.231290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:37993 (local address 127.27.153.126:39581)
0810 03:51:05.231401 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:05.231404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:05.231414 (+ 10us) server_negotiation.cc:407] Connection header received
0810 03:51:05.231444 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:05.231445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:05.231508 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:05.231616 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:05.232108 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.232629 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.233603 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.233817 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.233889 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:05.236715 (+ 2826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.67:37993: BlockingRecv error: recv got EOF from 127.27.153.67:37993 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 10 03:51:05 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:05 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:05.243165 12318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:05.237975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.67:46561)
0810 03:51:05.238157 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:05.238169 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:05.238248 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:05.238521 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:05.238524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:05.238684 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:05.238772 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:05.238777 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.239526 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.239530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:05.240181 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:05.240191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.240635 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.240638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:05.240703 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:05.241157 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:05.241169 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:05.243006 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98}
W20260810 03:51:05.243270 11830 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:05.238033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:46561 (local address 127.27.153.126:39581)
0810 03:51:05.238161 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:05.238165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:05.238220 (+ 55us) server_negotiation.cc:407] Connection header received
0810 03:51:05.238340 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:05.238342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:05.238385 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:05.238448 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:05.238878 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.239380 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.240299 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.240514 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.240591 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:05.243163 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.67:46561: BlockingRecv error: recv got EOF from 127.27.153.67:46561 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260810 03:51:05.243558 12316 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 03:51:05 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:05 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:05.850922 12321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:05.845304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.65:42945)
0810 03:51:05.845575 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:05.845589 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:05.845685 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:05.845990 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:05.845993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:05.846178 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:05.846302 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:05.846308 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.847156 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.847159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:05.847813 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:05.847823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.848346 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.848349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:05.848438 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:05.848956 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:05.848968 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:05.850796 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":82,"threads_started":1}
W20260810 03:51:05.851107 12327 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:05.845438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:42945 (local address 127.27.153.126:39581)
0810 03:51:05.845702 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:05.845706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:05.845721 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:51:05.845766 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:05.845768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:05.845812 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:05.845907 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:05.846446 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.847023 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.847943 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:05.848197 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:05.848290 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:05.850956 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.65:42945: BlockingRecv error: recv got EOF from 127.27.153.65:42945 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":77,"threads_started":1}
Aug 10 03:51:06 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:06 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:06.047348 12328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:06.041435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.66:49151)
0810 03:51:06.041720 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:06.041735 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:06.041858 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:06.042108 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:06.042110 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:06.042341 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:06.042458 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:06.042466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:06.043311 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:06.043315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:06.043990 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:06.044004 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:06.044479 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:06.044482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:06.044557 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:06.045160 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:06.045175 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:06.047214 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":96,"threads_started":1}
W20260810 03:51:06.047520 12327 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:06.041513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:49151 (local address 127.27.153.126:39581)
0810 03:51:06.041672 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:06.041676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:06.041823 (+ 147us) server_negotiation.cc:407] Connection header received
0810 03:51:06.041933 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:06.041936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:06.041985 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:06.042069 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:06.042594 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:06.043164 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:06.044097 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:06.044364 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:06.044467 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:06.047379 (+ 2912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.66:49151: BlockingRecv error: recv got EOF from 127.27.153.66:49151 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 10 03:51:06 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:06 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:06.249996 12331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:06.244269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.67:43787)
0810 03:51:06.244601 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:06.244616 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:06.244746 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:06.245026 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:06.245028 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:06.245239 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:06.245365 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:06.245372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:06.246173 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:06.246176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:06.246758 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:06.246768 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:06.247220 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:06.247224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:06.247325 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:06.247958 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:06.247971 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:06.249839 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":113,"threads_started":1}
W20260810 03:51:06.250106 12327 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:06.244405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:43787 (local address 127.27.153.126:39581)
0810 03:51:06.244589 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:06.244594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:06.244697 (+ 103us) server_negotiation.cc:407] Connection header received
0810 03:51:06.244848 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:06.244851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:06.244913 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:06.244982 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:06.245493 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:06.246046 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:06.246880 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:06.247109 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:06.247202 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:06.250004 (+ 2802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.67:43787: BlockingRecv error: recv got EOF from 127.27.153.67:43787 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 10 03:51:06 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:06 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:06.857882 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:06.851737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.65:38859)
0810 03:51:06.852045 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:06.852060 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:06.852161 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:06.852455 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:06.852458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:06.852681 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:06.852788 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:06.852794 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:06.853766 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:06.853770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:06.854376 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:06.854388 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:06.854958 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:06.854962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:06.855040 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:06.855717 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:06.855730 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:06.857697 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":91,"threads_started":1}
W20260810 03:51:06.858171 12333 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:06.851844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:38859 (local address 127.27.153.126:39581)
0810 03:51:06.852092 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:06.852096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:06.852115 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:51:06.852246 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:06.852248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:06.852296 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:06.852377 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:06.852914 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:06.853630 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:06.854534 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:06.854786 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:06.854879 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:06.858032 (+ 3153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.65:38859: BlockingRecv error: recv got EOF from 127.27.153.65:38859 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":84,"threads_started":1}
Aug 10 03:51:07 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:07 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:07.057741 12334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:07.048283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.66:60253)
0810 03:51:07.048649 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:07.048676 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:07.048799 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:07.049058 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:07.049061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:07.049306 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:07.049433 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:07.049443 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:07.050184 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:07.050187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:07.050795 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:07.050805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:07.051202 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:07.051205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:07.051271 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:07.055566 (+ 4295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:07.055582 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:07.057569 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":90,"threads_started":1}
W20260810 03:51:07.057909 12333 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:07.048383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:60253 (local address 127.27.153.126:39581)
0810 03:51:07.048552 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:07.048556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:07.048755 (+ 199us) server_negotiation.cc:407] Connection header received
0810 03:51:07.048884 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:07.048887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:07.048935 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:07.049027 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:07.049547 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:07.050074 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:07.050889 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:07.051103 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:07.051171 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:07.057769 (+ 6598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.66:60253: BlockingRecv error: recv got EOF from 127.27.153.66:60253 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 10 03:51:07 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:07 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:07.256703 12335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:07.250947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.67:53581)
0810 03:51:07.251267 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:07.251284 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:07.251392 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:07.251662 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:07.251664 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:07.251842 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:07.251944 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:07.251950 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:07.252696 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:07.252700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:07.253265 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:07.253280 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:07.253792 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:07.253795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:07.253868 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:07.254415 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:07.254427 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:07.256542 (+ 2115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":86,"threads_started":1}
W20260810 03:51:07.256814 12333 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:07.251173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:53581 (local address 127.27.153.126:39581)
0810 03:51:07.251328 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:07.251332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:07.251348 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:51:07.251504 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:07.251506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:07.251555 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:07.251626 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:07.252037 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:07.252565 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:07.253406 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:07.253650 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:07.253742 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:07.256698 (+ 2956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.67:53581: BlockingRecv error: recv got EOF from 127.27.153.67:53581 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 10 03:51:07 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:07 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:07.864535 12337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:07.858796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.65:60711)
0810 03:51:07.859070 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:07.859087 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:07.859194 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:07.859512 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:07.859516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:07.859726 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:07.859832 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:07.859839 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:07.860617 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:07.860621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:07.861223 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:07.861233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:07.861727 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:07.861730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:07.861805 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:07.862323 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:07.862336 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:07.864398 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":77,"threads_started":1}
W20260810 03:51:07.864704 12338 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:07.858902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:60711 (local address 127.27.153.126:39581)
0810 03:51:07.859176 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:07.859180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:07.859197 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:51:07.859279 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:07.859281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:07.859325 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:07.859394 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:07.859933 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:07.860486 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:07.861351 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:07.861590 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:07.861687 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:07.864561 (+ 2874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.65:60711: BlockingRecv error: recv got EOF from 127.27.153.65:60711 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":94,"threads_started":1}
Aug 10 03:51:08 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:08 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:08.064327 12339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:08.058643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.66:47893)
0810 03:51:08.058928 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:08.058942 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:08.059050 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:08.059331 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:08.059334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:08.059553 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:08.059706 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:08.059716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:08.060555 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:08.060558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:08.061148 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:08.061157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:08.061601 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:08.061604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:08.061676 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:08.062181 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:08.062192 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:08.064195 (+ 2003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":93,"threads_started":1}
W20260810 03:51:08.064508 12338 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:08.058781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:47893 (local address 127.27.153.126:39581)
0810 03:51:08.058913 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:08.058917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:08.059007 (+ 90us) server_negotiation.cc:407] Connection header received
0810 03:51:08.059128 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:08.059130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:08.059175 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:08.059259 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:08.059864 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:08.060423 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:08.061264 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:08.061486 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:08.061565 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:08.064364 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.66:47893: BlockingRecv error: recv got EOF from 127.27.153.66:47893 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 03:51:08 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:08 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:08.263682 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:08.257570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.67:41235)
0810 03:51:08.257855 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:08.257870 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:08.257971 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:08.258245 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:08.258247 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:08.258426 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:08.258531 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:08.258535 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:08.259348 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:08.259351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:08.260011 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:08.260023 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:08.260600 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:08.260603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:08.260679 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:08.261295 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:08.261312 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:08.263542 (+ 2230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":79,"threads_started":1}
W20260810 03:51:08.263852 12338 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:08.257710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:41235 (local address 127.27.153.126:39581)
0810 03:51:08.257903 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:08.257906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:08.257925 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:51:08.258060 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:08.258062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:08.258119 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:08.258218 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:08.258677 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:08.259222 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:08.260159 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:08.260421 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:08.260523 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:08.263717 (+ 3194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.67:41235: BlockingRecv error: recv got EOF from 127.27.153.67:41235 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 10 03:51:08 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:08 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:08.871811 12341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:08.865522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.65:55939)
0810 03:51:08.866252 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:08.866266 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:08.866385 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:08.866667 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:08.866670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:08.866890 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:08.867015 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:08.867024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:08.867972 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:08.867976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:08.868739 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:08.868752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:08.869205 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:08.869208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:08.869278 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:08.869812 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:08.869824 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:08.871687 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":507,"threads_started":1}
W20260810 03:51:08.872008 12342 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:08.865667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:55939 (local address 127.27.153.126:39581)
0810 03:51:08.866266 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:08.866270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:08.866346 (+ 76us) server_negotiation.cc:407] Connection header received
0810 03:51:08.866464 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:08.866467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:08.866513 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:08.866595 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:08.867179 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:08.867849 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:08.868865 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:08.869096 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:08.869181 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:08.871866 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.65:55939: BlockingRecv error: recv got EOF from 127.27.153.65:55939 (error 108)
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":343,"threads_started":1}
Aug 10 03:51:09 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:09 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:09.070307 12346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:09.065083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.66:41299)
0810 03:51:09.065346 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:09.065360 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:09.065463 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:09.065687 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:09.065689 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:09.065868 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:09.065971 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:09.065976 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:09.066688 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:09.066690 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:09.067256 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:09.067265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:09.067761 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:09.067764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:09.067833 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:09.068317 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:09.068330 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:09.070125 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":79,"threads_started":1}
W20260810 03:51:09.070398 12342 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:09.065180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:41299 (local address 127.27.153.126:39581)
0810 03:51:09.065315 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:09.065319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:09.065428 (+ 109us) server_negotiation.cc:407] Connection header received
0810 03:51:09.065535 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:09.065537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:09.065585 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:09.065661 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:09.066074 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:09.066606 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:09.067369 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:09.067637 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:09.067743 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:09.070277 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.66:41299: BlockingRecv error: recv got EOF from 127.27.153.66:41299 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 10 03:51:09 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:09 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:09.270335 12347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:09.264622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.67:56333)
0810 03:51:09.264972 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:09.264987 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:09.265125 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:09.265390 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:09.265393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:09.265596 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:09.265734 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:09.265740 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:09.266512 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:09.266516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:09.267177 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:09.267189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:09.267792 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:09.267797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:09.267886 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:09.268393 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:09.268406 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:09.270166 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":117,"threads_started":1}
W20260810 03:51:09.270437 12342 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:09.264769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:56333 (local address 127.27.153.126:39581)
0810 03:51:09.264962 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:09.264967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:09.265065 (+ 98us) server_negotiation.cc:407] Connection header received
0810 03:51:09.265223 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:09.265225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:09.265274 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:09.265348 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:09.265860 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:09.266398 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:09.267340 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:09.267673 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:09.267770 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:09.270329 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.67:56333: BlockingRecv error: recv got EOF from 127.27.153.67:56333 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 10 03:51:09 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:09 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:09.878587 12349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:09.872635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.65:41215)
0810 03:51:09.873035 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:09.873055 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:09.873188 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:09.873538 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:09.873541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:09.873735 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:09.873849 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:09.873855 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:09.874821 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:09.874824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:09.875563 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:09.875578 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:09.876091 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:09.876096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:09.876171 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:09.876690 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:09.876702 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:09.878447 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":89,"threads_started":1}
W20260810 03:51:09.878860 12350 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:09.872750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:41215 (local address 127.27.153.126:39581)
0810 03:51:09.873034 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:09.873040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:09.873138 (+ 98us) server_negotiation.cc:407] Connection header received
0810 03:51:09.873323 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:09.873326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:09.873375 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:09.873464 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:09.874020 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:09.874685 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:09.875699 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:09.875936 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:09.876030 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:09.878667 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.65:41215: BlockingRecv error: recv got EOF from 127.27.153.65:41215 (error 108)
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
Aug 10 03:51:10 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:10 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:10.076901 12351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:10.071278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.66:44695)
0810 03:51:10.071575 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:10.071588 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:10.071676 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:10.071906 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:10.071908 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:10.072120 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:10.072234 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:10.072240 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:10.073028 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:10.073032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:10.073614 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:10.073626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:10.074129 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:10.074134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:10.074223 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:10.074780 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:10.074791 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:10.076771 (+ 1980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":89,"threads_started":1}
W20260810 03:51:10.077090 12350 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:10.071490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:44695 (local address 127.27.153.126:39581)
0810 03:51:10.071615 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:10.071618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:10.071634 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:51:10.071755 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:10.071757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:10.071807 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:10.071882 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:10.072348 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:10.072899 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:10.073729 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:10.073990 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:10.074069 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:10.076957 (+ 2888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.66:44695: BlockingRecv error: recv got EOF from 127.27.153.66:44695 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 10 03:51:10 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:10 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:10.277050 12354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:10.271235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.67:36051)
0810 03:51:10.271585 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:10.271602 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:10.271728 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:10.271982 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:10.271985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:10.272187 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:10.272288 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:10.272295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:10.273058 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:10.273061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:10.273639 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:10.273648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:10.274082 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:10.274085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:10.274169 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:10.274737 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:10.274749 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:10.276904 (+ 2155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":92,"threads_started":1}
W20260810 03:51:10.277217 12350 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:10.271387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:36051 (local address 127.27.153.126:39581)
0810 03:51:10.271586 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:10.271591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:10.271682 (+ 91us) server_negotiation.cc:407] Connection header received
0810 03:51:10.271819 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:10.271822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:10.271872 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:10.271943 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:10.272407 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:10.272946 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:10.273753 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:10.273974 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:10.274063 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:10.277094 (+ 3031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.67:36051: BlockingRecv error: recv got EOF from 127.27.153.67:36051 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 10 03:51:10 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:10 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:10.885267 12355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:10.879518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.65:34039)
0810 03:51:10.879804 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:10.879821 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:10.879942 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:10.880228 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:10.880230 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:10.880415 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:10.880530 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:10.880536 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:10.881283 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:10.881286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:10.881872 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:10.881882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:10.882390 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:10.882393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:10.882487 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:10.883044 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:10.883059 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:10.885065 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":75,"threads_started":1}
W20260810 03:51:10.885421 12356 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:10.879643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:34039 (local address 127.27.153.126:39581)
0810 03:51:10.879941 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:10.879944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:10.879961 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:51:10.880028 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:10.880030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:10.880073 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:10.880154 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:10.880628 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:10.881168 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:10.882006 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:10.882258 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:10.882354 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:10.885282 (+ 2928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.65:34039: BlockingRecv error: recv got EOF from 127.27.153.65:34039 (error 108)
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":80,"threads_started":1}
Aug 10 03:51:11 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:11 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:11.083438 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:11.077722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.66:53253)
0810 03:51:11.078028 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:11.078041 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:11.078166 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:11.078449 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:11.078452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:11.078651 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:11.078753 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:11.078758 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:11.079602 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:11.079605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:11.080212 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:11.080222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:11.080707 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:11.080710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:11.080771 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:11.081264 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:11.081275 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:11.083310 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":87,"threads_started":1}
W20260810 03:51:11.083871 12356 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:11.077804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:53253 (local address 127.27.153.126:39581)
0810 03:51:11.077947 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:11.077952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:11.078132 (+ 180us) server_negotiation.cc:407] Connection header received
0810 03:51:11.078260 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:11.078264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:11.078322 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:11.078403 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:11.078897 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:11.079493 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:11.080343 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:11.080570 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:11.080681 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:11.083775 (+ 3094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.66:53253: BlockingRecv error: recv got EOF from 127.27.153.66:53253 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 10 03:51:11 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:11 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:11.283839 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:11.278075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.67:41691)
0810 03:51:11.278456 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:11.278469 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:11.278586 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:11.278857 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:11.278859 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:11.279061 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:11.279173 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:11.279180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:11.280008 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:11.280011 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:11.280579 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:11.280587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:11.281032 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:11.281036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:11.281151 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:11.281685 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:11.281697 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:11.283715 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":128,"threads_started":1}
W20260810 03:51:11.284087 12356 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:11.278187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:41691 (local address 127.27.153.126:39581)
0810 03:51:11.278379 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:11.278383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:11.278549 (+ 166us) server_negotiation.cc:407] Connection header received
0810 03:51:11.278689 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:11.278692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:11.278742 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:11.278821 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:11.279306 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:11.279897 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:11.280698 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:11.280922 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:11.281017 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:11.283947 (+ 2930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.67:41691: BlockingRecv error: recv got EOF from 127.27.153.67:41691 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 10 03:51:11 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:11 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:11.891897 12367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:11.886065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.65:44057)
0810 03:51:11.886413 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:11.886427 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:11.886533 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:11.886842 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:11.886844 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:11.887032 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:11.887131 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:11.887136 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:11.888037 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:11.888040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:11.888602 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:11.888612 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:11.889055 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:11.889058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:11.889119 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:11.889722 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:11.889735 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:11.891767 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":73,"threads_started":1}
W20260810 03:51:11.892115 12369 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:11.886163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:44057 (local address 127.27.153.126:39581)
0810 03:51:11.886430 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:11.886434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:11.886492 (+ 58us) server_negotiation.cc:407] Connection header received
0810 03:51:11.886629 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:11.886654 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:11.886713 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:11.886799 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:11.887288 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:11.887892 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:11.888728 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:11.888957 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:11.889037 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:11.891960 (+ 2923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.65:44057: BlockingRecv error: recv got EOF from 127.27.153.65:44057 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":91,"threads_started":1}
Aug 10 03:51:12 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:12 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:12.093492 12371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:12.084431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.66:54365)
0810 03:51:12.084795 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:12.084812 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:12.084922 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:12.085292 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:12.085294 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:12.085496 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:12.085608 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:12.085614 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:12.086476 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:12.086479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:12.087077 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:12.087087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:12.087640 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:12.087644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:12.087722 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:12.091243 (+ 3521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:12.091265 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:12.093331 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":82,"threads_started":1}
W20260810 03:51:12.093665 12369 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:12.084646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:54365 (local address 127.27.153.126:39581)
0810 03:51:12.084854 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:12.084858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:12.084876 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:51:12.085021 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:12.085024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:12.085081 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:12.085173 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:12.085773 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:12.086362 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:12.087219 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:12.087497 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:12.087585 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:12.093531 (+ 5946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.66:54365: BlockingRecv error: recv got EOF from 127.27.153.66:54365 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Aug 10 03:51:12 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:12 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:12.291023 12372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:12.284882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.67:35709)
0810 03:51:12.285220 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:12.285234 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:12.285361 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:12.285667 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:12.285670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:12.285858 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:12.285958 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:12.285966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:12.286947 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:12.286951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:12.287689 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:12.287702 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:12.288149 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:12.288152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:12.288218 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:12.288744 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:12.288757 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:12.290896 (+ 2139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":87,"threads_started":1}
W20260810 03:51:12.291285 12369 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:12.285024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:35709 (local address 127.27.153.126:39581)
0810 03:51:12.285214 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:12.285217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:12.285349 (+ 132us) server_negotiation.cc:407] Connection header received
0810 03:51:12.285457 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:12.285460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:12.285512 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:12.285577 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:12.286103 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:12.286832 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:12.287800 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:12.288037 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:12.288118 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:12.291065 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.67:35709: BlockingRecv error: recv got EOF from 127.27.153.67:35709 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 10 03:51:12 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:12 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:12.898952 12373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:12.892914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.65:53021)
0810 03:51:12.893241 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:12.893255 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:12.893352 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:12.893684 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:12.893687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:12.893871 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:12.893976 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:12.893982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:12.894857 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:12.894860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:12.895440 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:12.895450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:12.896012 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:12.896016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:12.896087 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:12.896746 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:12.896760 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:12.898786 (+ 2026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":93,"threads_started":1}
W20260810 03:51:12.899129 12375 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:12.893138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:53021 (local address 127.27.153.126:39581)
0810 03:51:12.893387 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:12.893391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:12.893404 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:51:12.893452 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:12.893454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:12.893500 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:12.893619 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:12.894138 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:12.894719 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:12.895627 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:12.895884 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:12.895986 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:12.898988 (+ 3002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.65:53021: BlockingRecv error: recv got EOF from 127.27.153.65:53021 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":68,"threads_started":1}
Aug 10 03:51:13 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:13 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:13.101002 12376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:13.094622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.66:57847)
0810 03:51:13.094935 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:13.094952 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:13.095069 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:13.095483 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:13.095486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:13.095702 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:13.095814 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:13.095821 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:13.096977 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:13.096981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:13.097777 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:13.097791 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:13.098272 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:13.098275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:13.098348 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:13.098947 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:13.098959 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:13.100871 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":101,"threads_started":1}
W20260810 03:51:13.101212 12375 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:13.094783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:57847 (local address 127.27.153.126:39581)
0810 03:51:13.094962 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:13.094965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:13.095035 (+ 70us) server_negotiation.cc:407] Connection header received
0810 03:51:13.095222 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:13.095226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:13.095280 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:13.095372 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:13.095981 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:13.096809 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:13.097902 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:13.098132 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:13.098216 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:13.101083 (+ 2867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.66:57847: BlockingRecv error: recv got EOF from 127.27.153.66:57847 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 10 03:51:13 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:13 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:13.297804 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:13.291945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.67:56999)
0810 03:51:13.292272 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:13.292288 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:13.292413 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:13.292716 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:13.292719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:13.292945 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:13.293052 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:13.293059 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:13.293973 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:13.293977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:13.294545 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:13.294554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:13.295023 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:13.295026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:13.295113 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:13.295646 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:13.295661 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:13.297669 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":95,"threads_started":1}
W20260810 03:51:13.298036 12375 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:13.292081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:56999 (local address 127.27.153.126:39581)
0810 03:51:13.292250 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:13.292254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:13.292367 (+ 113us) server_negotiation.cc:407] Connection header received
0810 03:51:13.292521 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:13.292524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:13.292583 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:13.292705 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:13.293192 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:13.293858 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:13.294683 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:13.294900 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:13.294989 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:13.297889 (+ 2900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.67:56999: BlockingRecv error: recv got EOF from 127.27.153.67:56999 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 03:51:13 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:13 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:13.905989 12379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:13.900001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.65:40071)
0810 03:51:13.900422 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:13.900439 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:13.900547 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:13.900838 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:13.900841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:13.901050 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:13.901171 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:13.901180 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:13.902191 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:13.902194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:13.902779 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:13.902788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:13.903271 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:13.903274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:13.903345 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:13.903871 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:13.903884 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:13.905852 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":89,"threads_started":1}
W20260810 03:51:13.906138 12380 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:13.900173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:40071 (local address 127.27.153.126:39581)
0810 03:51:13.900402 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:13.900407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:13.900518 (+ 111us) server_negotiation.cc:407] Connection header received
0810 03:51:13.900639 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:13.900641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:13.900690 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:13.900784 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:13.901342 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:13.902070 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:13.902912 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:13.903149 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:13.903240 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:13.906013 (+ 2773us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.65:40071: BlockingRecv error: recv got EOF from 127.27.153.65:40071 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":66,"threads_started":1}
Aug 10 03:51:14 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:14 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:14.107862 12383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:14.101957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.66:40063)
0810 03:51:14.102239 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:14.102253 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:14.102365 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:14.102699 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:14.102702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:14.102904 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:14.103019 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:14.103026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:14.103901 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:14.103905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:14.104492 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:14.104503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:14.105020 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:14.105024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:14.105109 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:14.105682 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:14.105695 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:14.107690 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":86,"threads_started":1}
W20260810 03:51:14.107952 12380 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:14.102109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:40063 (local address 127.27.153.126:39581)
0810 03:51:14.102294 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:14.102298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:14.102316 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:51:14.102471 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:14.102474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:14.102522 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:14.102604 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:14.103180 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:14.103765 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:14.104648 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:14.104882 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:14.104976 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:14.107847 (+ 2871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.66:40063: BlockingRecv error: recv got EOF from 127.27.153.66:40063 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 10 03:51:14 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:14 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:14.305011 12385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:14.298717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.67:50227)
0810 03:51:14.299081 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:14.299097 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:14.299222 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:14.299540 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:14.299543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:14.299783 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:14.299922 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:14.299931 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:14.300929 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:14.300932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:14.301505 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:14.301514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:14.302054 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:14.302058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:14.302136 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:14.302749 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:14.302762 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:14.304849 (+ 2087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":83,"threads_started":1}
W20260810 03:51:14.305380 12380 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:14.298824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:50227 (local address 127.27.153.126:39581)
0810 03:51:14.298991 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:14.298996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:14.299183 (+ 187us) server_negotiation.cc:407] Connection header received
0810 03:51:14.299326 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:14.299329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:14.299388 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:14.299502 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:14.300190 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:14.300803 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:14.301648 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:14.301901 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:14.301995 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:14.305252 (+ 3257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.67:50227: BlockingRecv error: recv got EOF from 127.27.153.67:50227 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 03:51:14 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:14 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:14.913421 12386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:14.907090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.65:50917)
0810 03:51:14.907403 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:14.907682 (+ 279us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:14.907806 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:14.908082 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:14.908085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:14.908319 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:14.908449 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:14.908458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:14.909423 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:14.909426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:14.910061 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:14.910072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:14.910606 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:14.910609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:14.910679 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:14.911200 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:14.911213 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:14.913279 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":96,"threads_started":1}
W20260810 03:51:14.913618 12391 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:14.907227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:50917 (local address 127.27.153.126:39581)
0810 03:51:14.907534 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:14.907538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:14.907859 (+ 321us) server_negotiation.cc:407] Connection header received
0810 03:51:14.907898 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:14.907900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:14.907945 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:14.908028 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:14.908605 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:14.909313 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:14.910187 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:14.910456 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:14.910546 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:14.913479 (+ 2933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.65:50917: BlockingRecv error: recv got EOF from 127.27.153.65:50917 (error 108)
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":82,"threads_started":1}
Aug 10 03:51:15 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:15 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:15.114791 12392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:15.108895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.66:49633)
0810 03:51:15.109171 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:15.109186 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:15.109281 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:15.109569 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:15.109572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:15.109785 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:15.109895 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:15.109903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:15.110711 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:15.110715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:15.111289 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:15.111298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:15.111843 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:15.111846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:15.111919 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:15.112501 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:15.112514 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:15.114632 (+ 2118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":76,"threads_started":1}
W20260810 03:51:15.114903 12391 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:15.109080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:49633 (local address 127.27.153.126:39581)
0810 03:51:15.109219 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:15.109223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:15.109239 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:51:15.109365 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:15.109368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:15.109414 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:15.109494 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:15.110002 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:15.110552 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:15.111443 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:15.111710 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:15.111806 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:15.114800 (+ 2994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.66:49633: BlockingRecv error: recv got EOF from 127.27.153.66:49633 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 03:51:15 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:15 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:15.312093 12394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:15.306136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.67:43093)
0810 03:51:15.306481 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:15.306498 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:15.306653 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:15.306957 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:15.306960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:15.307180 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:15.307324 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:15.307333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:15.308206 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:15.308209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:15.308776 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:15.308785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:15.309240 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:15.309243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:15.309311 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:15.309849 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:15.309860 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:15.311931 (+ 2071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":92,"threads_started":1}
W20260810 03:51:15.312218 12391 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:15.306350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:43093 (local address 127.27.153.126:39581)
0810 03:51:15.306477 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:15.306482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:15.306592 (+ 110us) server_negotiation.cc:407] Connection header received
0810 03:51:15.306761 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:15.306764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:15.306822 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:15.306907 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:15.307507 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:15.308102 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:15.308904 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:15.309130 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:15.309215 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:15.312110 (+ 2895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.67:43093: BlockingRecv error: recv got EOF from 127.27.153.67:43093 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 10 03:51:15 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:15 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:15.920008 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:15.914283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.65:40757)
0810 03:51:15.914568 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:15.914582 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:15.914675 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:15.914929 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:15.914931 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:15.915110 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:15.915211 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:15.915217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:15.916042 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:15.916045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:15.916632 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:15.916643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:15.917156 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:15.917159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:15.917230 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:15.917822 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:15.917834 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:15.919839 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":91,"threads_started":1}
W20260810 03:51:15.920151 12398 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:15.914411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:40757 (local address 127.27.153.126:39581)
0810 03:51:15.914647 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:15.914650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:15.914664 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:51:15.914779 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:15.914782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:15.914826 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:15.914900 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:15.915335 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:15.915929 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:15.916779 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:15.917029 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:15.917136 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:15.920028 (+ 2892us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.65:40757: BlockingRecv error: recv got EOF from 127.27.153.65:40757 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":73,"threads_started":1}
Aug 10 03:51:16 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:16 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:16.121711 12399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:16.115945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.66:59129)
0810 03:51:16.116233 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:16.116248 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:16.116350 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:16.116689 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:16.116692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:16.116874 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:16.116974 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:16.116979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:16.117800 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:16.117804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:16.118394 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:16.118405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:16.118925 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:16.118928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:16.119000 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:16.119541 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:16.119554 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:16.121558 (+ 2004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":84,"threads_started":1}
W20260810 03:51:16.121896 12398 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:16.116110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:59129 (local address 127.27.153.126:39581)
0810 03:51:16.116292 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:16.116296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:16.116315 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:51:16.116493 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:16.116497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:16.116552 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:16.116636 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:16.117124 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:16.117671 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:16.118488 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:16.118796 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:16.118904 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:16.121768 (+ 2864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.66:59129: BlockingRecv error: recv got EOF from 127.27.153.66:59129 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 10 03:51:16 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:16 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:16.318818 12401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:16.313236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.67:39397)
0810 03:51:16.313536 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:16.313552 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:16.313679 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:16.313957 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:16.313961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:16.314160 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:16.314271 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:16.314278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:16.315055 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:16.315058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:16.315666 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:16.315675 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:16.316152 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:16.316156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:16.316242 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:16.316802 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:16.316815 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:16.318649 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":96,"threads_started":1}
W20260810 03:51:16.318931 12398 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:16.313388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:39397 (local address 127.27.153.126:39581)
0810 03:51:16.313538 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:16.313543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:16.313616 (+ 73us) server_negotiation.cc:407] Connection header received
0810 03:51:16.313787 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:16.313790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:16.313842 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:16.313910 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:16.314407 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:16.314936 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:16.315801 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:16.316033 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:16.316126 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:16.318823 (+ 2697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.67:39397: BlockingRecv error: recv got EOF from 127.27.153.67:39397 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 03:51:16 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:16 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:16.926996 12402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:16.921085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.65:56003)
0810 03:51:16.921385 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:16.921400 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:16.921502 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:16.921802 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:16.921805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:16.921991 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:16.922113 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:16.922120 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:16.922893 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:16.922897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:16.923548 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:16.923562 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:16.924110 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:16.924113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:16.924204 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:16.924838 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:16.924851 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:16.926872 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":87,"threads_started":1}
W20260810 03:51:16.927170 12404 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:16.921267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:56003 (local address 127.27.153.126:39581)
0810 03:51:16.921552 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:16.921555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:16.921570 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:51:16.921615 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:16.921617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:16.921659 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:16.921767 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:16.922222 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:16.922773 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:16.923707 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:16.923951 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:16.924074 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:16.927024 (+ 2950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.65:56003: BlockingRecv error: recv got EOF from 127.27.153.65:56003 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":79,"threads_started":1}
Aug 10 03:51:17 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:17 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:17.133287 12406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:17.122456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.66:56159)
0810 03:51:17.122764 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:17.122779 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:17.122878 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:17.123305 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:17.123308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:17.123560 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:17.123673 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:17.123680 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:17.124497 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:17.124500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:17.125408 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:17.125424 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:17.126109 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:17.126113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:17.126218 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:17.130953 (+ 4735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:17.130975 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:17.133094 (+ 2119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":90,"threads_started":1}
W20260810 03:51:17.133373 12404 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:17.122644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:56159 (local address 127.27.153.126:39581)
0810 03:51:17.123009 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:17.123014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:17.123030 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:51:17.123072 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:17.123074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:17.123123 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:17.123201 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:17.123831 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:17.124384 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:17.125585 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:17.125963 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:17.126085 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:17.133267 (+ 7182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.66:56159: BlockingRecv error: recv got EOF from 127.27.153.66:56159 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 10 03:51:17 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:17 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:17.325697 12407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:17.319755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.67:55083)
0810 03:51:17.320100 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:17.320117 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:17.320248 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:17.320529 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:17.320532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:17.320757 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:17.320863 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:17.320870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:17.321669 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:17.321673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:17.322234 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:17.322243 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:17.322715 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:17.322719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:17.322808 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:17.323308 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:17.323320 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:17.325532 (+ 2212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":115,"threads_started":1}
W20260810 03:51:17.325981 12404 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:17.319907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:55083 (local address 127.27.153.126:39581)
0810 03:51:17.320095 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:17.320099 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:17.320208 (+ 109us) server_negotiation.cc:407] Connection header received
0810 03:51:17.320354 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:17.320357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:17.320407 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:17.320476 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:17.321001 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:17.321534 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:17.322366 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:17.322588 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:17.322677 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:17.325857 (+ 3180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.67:55083: BlockingRecv error: recv got EOF from 127.27.153.67:55083 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 10 03:51:17 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:17 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:17.933785 12409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:17.927895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.65:39491)
0810 03:51:17.928176 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:17.928191 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:17.928288 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:17.928555 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:17.928558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:17.928767 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:17.928872 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:17.928878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:17.929705 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:17.929707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:17.930262 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:17.930271 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:17.930813 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:17.930817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:17.930890 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:17.931433 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:17.931446 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:17.933596 (+ 2150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":81,"threads_started":1}
W20260810 03:51:17.933885 12410 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:17.928031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:39491 (local address 127.27.153.126:39581)
0810 03:51:17.928310 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:17.928314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:17.928329 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:51:17.928370 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:17.928372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:17.928414 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:17.928503 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:17.928994 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:17.929561 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:17.930430 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:17.930670 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:17.930760 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:17.933764 (+ 3004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.65:39491: BlockingRecv error: recv got EOF from 127.27.153.65:39491 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":84,"threads_started":1}
Aug 10 03:51:18 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:18 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:18.139878 12411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:18.134244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.66:58799)
0810 03:51:18.134575 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:18.134590 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:18.134704 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:18.134939 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:18.134941 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:18.135130 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:18.135246 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:18.135253 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:18.136134 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:18.136137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:18.136732 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:18.136742 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:18.137231 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:18.137233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:18.137292 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:18.137860 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:18.137872 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:18.139733 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":92,"threads_started":1}
W20260810 03:51:18.139998 12410 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:18.134423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:58799 (local address 127.27.153.126:39581)
0810 03:51:18.134589 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:18.134592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:18.134679 (+ 87us) server_negotiation.cc:407] Connection header received
0810 03:51:18.134788 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:18.134791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:18.134836 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:18.134907 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:18.135421 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:18.136016 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:18.136887 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:18.137123 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:18.137217 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:18.139891 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.66:58799: BlockingRecv error: recv got EOF from 127.27.153.66:58799 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 10 03:51:18 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:18 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:18.332578 12413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:18.326671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.67:36233)
0810 03:51:18.326979 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:18.326996 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:18.327098 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:18.327420 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:18.327423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:18.327664 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:18.327768 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:18.327775 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:18.328544 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:18.328548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:18.329118 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:18.329127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:18.329633 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:18.329636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:18.329714 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:18.330307 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:18.330322 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:18.332450 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":90,"threads_started":1}
W20260810 03:51:18.332899 12410 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:18.326805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:36233 (local address 127.27.153.126:39581)
0810 03:51:18.326958 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:18.326963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:18.327067 (+ 104us) server_negotiation.cc:407] Connection header received
0810 03:51:18.327199 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:18.327201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:18.327251 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:18.327347 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:18.327881 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:18.328416 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:18.329247 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:18.329479 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:18.329561 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:18.332674 (+ 3113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.67:36233: BlockingRecv error: recv got EOF from 127.27.153.67:36233 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 10 03:51:18 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:18 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:18.940774 12414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:18.934795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.65:56833)
0810 03:51:18.935107 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:18.935122 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:18.935219 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:18.935588 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:18.935590 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:18.935801 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:18.935904 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:18.935909 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:18.936914 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:18.936917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:18.937574 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:18.937585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:18.938053 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:18.938056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:18.938141 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:18.938676 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:18.938688 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:18.940626 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":83,"threads_started":1}
W20260810 03:51:18.940948 12416 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:18.934958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:56833 (local address 127.27.153.126:39581)
0810 03:51:18.935289 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:18.935292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:18.935307 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:51:18.935351 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:18.935353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:18.935396 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:18.935515 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:18.936076 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:18.936793 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:18.937703 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:18.937937 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:18.938027 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:18.940805 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.65:56833: BlockingRecv error: recv got EOF from 127.27.153.65:56833 (error 108)
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":73,"threads_started":1}
Aug 10 03:51:19 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:19 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:19.147096 12417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:19.140815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.66:55605)
0810 03:51:19.141091 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:19.141125 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:19.141253 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:19.141584 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:19.141589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:19.141813 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:19.141921 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:19.141929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:19.142890 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:19.142894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:19.143717 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:19.143729 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:19.144262 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:19.144265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:19.144335 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:19.144890 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:19.144902 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:19.146947 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":81,"threads_started":1}
W20260810 03:51:19.147293 12416 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:19.140958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:55605 (local address 127.27.153.126:39581)
0810 03:51:19.141124 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:19.141129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:19.141210 (+ 81us) server_negotiation.cc:407] Connection header received
0810 03:51:19.141348 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:19.141352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:19.141412 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:19.141492 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:19.142049 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:19.142752 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:19.143869 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:19.144128 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:19.144239 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:19.147164 (+ 2925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.66:55605: BlockingRecv error: recv got EOF from 127.27.153.66:55605 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 03:51:19 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:19 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.67@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:19.339262 12418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:19.333458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.67:34089)
0810 03:51:19.333759 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:19.333773 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:19.333867 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:19.334173 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:19.334175 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:19.334364 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:19.334467 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:19.334473 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:19.335280 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:19.335283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:19.335914 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:19.335925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:19.336446 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:19.336450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:19.336549 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:19.337131 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:19.337144 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:19.339113 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":81,"threads_started":1}
W20260810 03:51:19.339388 12416 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:19.333616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:34089 (local address 127.27.153.126:39581)
0810 03:51:19.333800 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:19.333805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:19.333838 (+ 33us) server_negotiation.cc:407] Connection header received
0810 03:51:19.334002 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:19.334005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:19.334066 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:19.334144 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:19.334593 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:19.335162 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:19.336030 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:19.336333 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:19.336422 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:19.339266 (+ 2844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.67:34089: BlockingRecv error: recv got EOF from 127.27.153.67:34089 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 10 03:51:19 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:19 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.65@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:19.947916 12420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:19.941870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.65:46053)
0810 03:51:19.942184 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:19.942241 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:19.942355 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:19.942754 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:19.942757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:19.942947 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:19.943049 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:19.943055 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:19.944014 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:19.944017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:19.944681 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:19.944692 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:19.945146 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:19.945149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:19.945223 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:19.945767 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:19.945780 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:19.947790 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":97,"threads_started":1}
W20260810 03:51:19.948118 12421 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:19.942006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:46053 (local address 127.27.153.126:39581)
0810 03:51:19.942314 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:19.942318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:19.942334 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:51:19.942437 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:19.942439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:19.942486 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:19.942585 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:19.943152 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:19.943896 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:19.944785 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:19.945013 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:19.945091 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:19.947972 (+ 2881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.65:46053: BlockingRecv error: recv got EOF from 127.27.153.65:46053 (error 108)
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":76,"threads_started":1}
Aug 10 03:51:20 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 03:51:20 dist-test-slave-pww0 krb5kdc[10446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.153.66@KRBTEST.COM for oryx/127.27.153.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 03:51:20.154268 12422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:20.148175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39581 (local address 127.27.153.66:56671)
0810 03:51:20.148558 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:20.148571 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:20.148706 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:20.148999 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:20.149002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:20.149255 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:20.149384 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:20.149391 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:20.150167 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:20.150170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:20.150759 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:20.150768 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:20.151252 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:20.151255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:20.151325 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:20.151901 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:20.151913 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:20.154122 (+ 2209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.153.126:39581: Server oryx/127.27.153.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":83,"threads_started":1}
W20260810 03:51:20.154466 12421 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:20.148328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:56671 (local address 127.27.153.126:39581)
0810 03:51:20.148461 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:20.148466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:20.148684 (+ 218us) server_negotiation.cc:407] Connection header received
0810 03:51:20.148806 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:20.148809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:20.148864 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:20.148948 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:20.149505 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:20.150043 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:20.150895 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:20.151132 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:20.151214 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:20.154311 (+ 3097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.153.66:56671: BlockingRecv error: recv got EOF from 127.27.153.66:56671 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 03:51:20.233031 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 11532
I20260810 03:51:20.246366 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 11832
I20260810 03:51:20.259174 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 12059
I20260810 03:51:20.272128 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 11396
2026-08-10T03:51:20Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18366 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T03:51:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:51:20Z Disabled control of system clock
I20260810 03:51:20.317917 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:42171
--webserver_interface=127.27.153.126
--webserver_port=0
--builtin_ntp_servers=127.27.153.84:34027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:42171
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 03:51:20.435050 12429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:51:20.435343 12429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:51:20.435379 12429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:51:20.439818 12429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:51:20.439913 12429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:51:20.439927 12429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:51:20.439944 12429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:51:20.439958 12429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:51:20.445214 12429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:34027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:42171
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:42171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:51:20.446451 12429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:51:20.447945 12429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:51:20.457432 12439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:20.457434 12435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:20.457434 12434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:20.457480 12429 server_base.cc:1034] running on GCE node
I20260810 03:51:20.458415 12429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:51:20.459666 12429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:51:20.460850 12429 hybrid_clock.cc:648] HybridClock initialized: now 1786333880460833 us; error 35 us; skew 500 ppm
I20260810 03:51:20.462929 12429 webserver.cc:460] Webserver started at http://127.27.153.126:41299/ using document root <none> and password file <none>
I20260810 03:51:20.463604 12429 fs_manager.cc:359] Metadata directory not provided
I20260810 03:51:20.463680 12429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:51:20.463904 12429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:51:20.466151 12429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "543c720e26d6401aaf9960e8b3c94afa"
format_stamp: "Formatted at 2026-08-10 03:51:20 on dist-test-slave-pww0"
server_key: "1497ba835a29051168cb139a19a9b00c"
server_key_iv: "10117c0f3d19b8aa49cd75dbb70cdc50"
server_key_version: "encryptionkey@0"
I20260810 03:51:20.466790 12429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "543c720e26d6401aaf9960e8b3c94afa"
format_stamp: "Formatted at 2026-08-10 03:51:20 on dist-test-slave-pww0"
server_key: "1497ba835a29051168cb139a19a9b00c"
server_key_iv: "10117c0f3d19b8aa49cd75dbb70cdc50"
server_key_version: "encryptionkey@0"
I20260810 03:51:20.470831 12429 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:51:20.473714 12455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:20.475144 12429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:51:20.475287 12429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "543c720e26d6401aaf9960e8b3c94afa"
format_stamp: "Formatted at 2026-08-10 03:51:20 on dist-test-slave-pww0"
server_key: "1497ba835a29051168cb139a19a9b00c"
server_key_iv: "10117c0f3d19b8aa49cd75dbb70cdc50"
server_key_version: "encryptionkey@0"
I20260810 03:51:20.475431 12429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:51:20.524152 12429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:51:20.524967 12429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:51:20.525159 12429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:51:20.532851 12429 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:42171
I20260810 03:51:20.532876 12543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:42171 every 8 connection(s)
I20260810 03:51:20.533944 12429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:51:20.535775 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 12429
I20260810 03:51:20.535888 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:51:20.536166 28261 external_mini_cluster.cc:1428] Setting key 3ebd90a970032f3b42e139b033839a26
I20260810 03:51:20.537891 12544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:51:20.545488 12544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 543c720e26d6401aaf9960e8b3c94afa: Bootstrap starting.
I20260810 03:51:20.547570 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:20.537734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56824 (local address 127.27.153.126:42171)
0810 03:51:20.538486 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:20.538506 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:20.538551 (+ 45us) server_negotiation.cc:407] Connection header received
0810 03:51:20.539442 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:20.539502 (+ 60us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:20.540040 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:20.540467 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:20.540953 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:20.542444 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:20.543484 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:20.544255 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:20.544432 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:20.545335 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:51:20.545342 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:51:20.545360 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:51:20.545876 (+ 516us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:51:20.545954 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:51:20.545983 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:51:20.546078 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:20.546253 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 03:51:20.546540 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":126,"threads_started":1}
I20260810 03:51:20.548403 12544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 543c720e26d6401aaf9960e8b3c94afa: Neither blocks nor log segments found. Creating new log.
I20260810 03:51:20.549320 12544 log.cc:824] T 00000000000000000000000000000000 P 543c720e26d6401aaf9960e8b3c94afa: Log is configured to *not* fsync() on all Append() calls
I20260810 03:51:20.551576 12544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 543c720e26d6401aaf9960e8b3c94afa: No bootstrap required, opened a new log
I20260810 03:51:20.555215 12544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 543c720e26d6401aaf9960e8b3c94afa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543c720e26d6401aaf9960e8b3c94afa" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42171 } }
I20260810 03:51:20.555539 12544 raft_consensus.cc:374] T 00000000000000000000000000000000 P 543c720e26d6401aaf9960e8b3c94afa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:51:20.555619 12544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 543c720e26d6401aaf9960e8b3c94afa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 543c720e26d6401aaf9960e8b3c94afa, State: Initialized, Role: FOLLOWER
I20260810 03:51:20.556099 12544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 543c720e26d6401aaf9960e8b3c94afa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543c720e26d6401aaf9960e8b3c94afa" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42171 } }
I20260810 03:51:20.556242 12544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 543c720e26d6401aaf9960e8b3c94afa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:51:20.556308 12544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 543c720e26d6401aaf9960e8b3c94afa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:51:20.556406 12544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 543c720e26d6401aaf9960e8b3c94afa [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:51:20.557441 12544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 543c720e26d6401aaf9960e8b3c94afa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543c720e26d6401aaf9960e8b3c94afa" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42171 } }
I20260810 03:51:20.557802 12544 leader_election.cc:302] T 00000000000000000000000000000000 P 543c720e26d6401aaf9960e8b3c94afa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 543c720e26d6401aaf9960e8b3c94afa; no voters:
I20260810 03:51:20.558133 12544 leader_election.cc:288] T 00000000000000000000000000000000 P 543c720e26d6401aaf9960e8b3c94afa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:51:20.558264 12552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 543c720e26d6401aaf9960e8b3c94afa [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:51:20.558499 12552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 543c720e26d6401aaf9960e8b3c94afa [term 1 LEADER]: Becoming Leader. State: Replica: 543c720e26d6401aaf9960e8b3c94afa, State: Running, Role: LEADER
I20260810 03:51:20.558852 12552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 543c720e26d6401aaf9960e8b3c94afa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543c720e26d6401aaf9960e8b3c94afa" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42171 } }
I20260810 03:51:20.559580 12544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 543c720e26d6401aaf9960e8b3c94afa [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:51:20.560757 12559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 543c720e26d6401aaf9960e8b3c94afa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 543c720e26d6401aaf9960e8b3c94afa. Latest consensus state: current_term: 1 leader_uuid: "543c720e26d6401aaf9960e8b3c94afa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543c720e26d6401aaf9960e8b3c94afa" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42171 } } }
I20260810 03:51:20.560909 12559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 543c720e26d6401aaf9960e8b3c94afa [sys.catalog]: This master's current role is: LEADER
I20260810 03:51:20.560761 12554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 543c720e26d6401aaf9960e8b3c94afa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "543c720e26d6401aaf9960e8b3c94afa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "543c720e26d6401aaf9960e8b3c94afa" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42171 } } }
I20260810 03:51:20.561136 12554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 543c720e26d6401aaf9960e8b3c94afa [sys.catalog]: This master's current role is: LEADER
I20260810 03:51:20.561331 12570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:51:20.564273 12570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:51:20.570531 12570 catalog_manager.cc:1347] Generated new cluster ID: 26d6a6ce710a485a9ee26a0f5a9de54e
I20260810 03:51:20.570633 12570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:51:20.585309 12570 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:51:20.586661 12570 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:51:20.600481 12570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 543c720e26d6401aaf9960e8b3c94afa: Generated new TSK 0
I20260810 03:51:20.601315 12570 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:51:20.608752 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:42171
--builtin_ntp_servers=127.27.153.84:34027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 03:51:20.736629 12593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:51:20.736934 12593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:51:20.736974 12593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:51:20.741143 12593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:51:20.741215 12593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:51:20.741271 12593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:51:20.746206 12593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:34027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:42171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:51:20.747349 12593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:51:20.748813 12593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:51:20.758812 12598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:20.758888 12600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:20.758802 12603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:20.759112 12593 server_base.cc:1034] running on GCE node
I20260810 03:51:20.759697 12593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:51:20.760502 12593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:51:20.761665 12593 hybrid_clock.cc:648] HybridClock initialized: now 1786333880761645 us; error 32 us; skew 500 ppm
I20260810 03:51:20.763870 12593 webserver.cc:460] Webserver started at http://127.27.153.65:35159/ using document root <none> and password file <none>
I20260810 03:51:20.764499 12593 fs_manager.cc:359] Metadata directory not provided
I20260810 03:51:20.764568 12593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:51:20.764778 12593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:51:20.767042 12593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "14601088620845c7b67fa93468643a5e"
format_stamp: "Formatted at 2026-08-10 03:51:20 on dist-test-slave-pww0"
server_key: "b821624d7761713c081da49d5bd54a94"
server_key_iv: "2a8d68f4f4a12fe7fe6d84f8a4218391"
server_key_version: "encryptionkey@0"
I20260810 03:51:20.767689 12593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "14601088620845c7b67fa93468643a5e"
format_stamp: "Formatted at 2026-08-10 03:51:20 on dist-test-slave-pww0"
server_key: "b821624d7761713c081da49d5bd54a94"
server_key_iv: "2a8d68f4f4a12fe7fe6d84f8a4218391"
server_key_version: "encryptionkey@0"
I20260810 03:51:20.771808 12593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:51:20.774714 12617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:20.776208 12593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:51:20.776345 12593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "14601088620845c7b67fa93468643a5e"
format_stamp: "Formatted at 2026-08-10 03:51:20 on dist-test-slave-pww0"
server_key: "b821624d7761713c081da49d5bd54a94"
server_key_iv: "2a8d68f4f4a12fe7fe6d84f8a4218391"
server_key_version: "encryptionkey@0"
I20260810 03:51:20.776475 12593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:51:20.790405 12593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:51:20.791144 12593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:51:20.791342 12593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:51:20.791994 12593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:51:20.792971 12593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:51:20.793020 12593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:20.793077 12593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:51:20.793108 12593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:20.801736 12593 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:40085
I20260810 03:51:20.801780 12821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:40085 every 8 connection(s)
I20260810 03:51:20.802667 12593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:51:20.807500 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 12593
I20260810 03:51:20.807618 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:51:20.807914 28261 external_mini_cluster.cc:1428] Setting key 920b48675d4b5b1622378eb771ff60be
I20260810 03:51:20.809991 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:42171
--builtin_ntp_servers=127.27.153.84:34027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 03:51:20.812569 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:20.804506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:38709 (local address 127.27.153.126:42171)
0810 03:51:20.804643 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:20.804647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:20.805748 (+ 1101us) server_negotiation.cc:407] Connection header received
0810 03:51:20.807182 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:20.807185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:20.807254 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:20.807332 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:20.808448 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:20.809137 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:20.810404 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:20.810704 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:20.810806 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:20.812017 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:51:20.812021 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:51:20.812023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:51:20.812040 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:51:20.812061 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:51:20.812070 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:51:20.812147 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:20.812365 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 03:51:20.812449 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 03:51:20.813668 12826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:20.804865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42171 (local address 127.27.153.65:38709)
0810 03:51:20.805553 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:20.805595 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:20.806949 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:20.807562 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:20.807571 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:20.807688 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:20.808243 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:20.808264 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:20.809280 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:20.809284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:20.810263 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:20.810274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:20.810849 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:20.810852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:20.810998 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:20.811659 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:51:20.811662 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:20.811726 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:51:20.811824 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:51:20.812156 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:51:20.812172 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:51:20.812177 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 03:51:20.812314 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 03:51:20.812643 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":121,"threads_started":1}
I20260810 03:51:20.815085 12822 heartbeater.cc:347] Connected to a master server at 127.27.153.126:42171
I20260810 03:51:20.815346 12822 heartbeater.cc:464] Registering TS with master...
I20260810 03:51:20.815935 12822 heartbeater.cc:511] Master 127.27.153.126:42171 requested a full tablet report, sending...
I20260810 03:51:20.817508 12480 ts_manager.cc:194] Registered new tserver with Master: 14601088620845c7b67fa93468643a5e (127.27.153.65:40085)
I20260810 03:51:20.818735 12480 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.153.65:38709
W20260810 03:51:20.926844 12831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:51:20.927351 12831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:51:20.927580 12831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:51:20.932163 12831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:51:20.932247 12831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:51:20.932328 12831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:51:20.937934 12831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:34027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:42171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:51:20.939159 12831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:51:20.940691 12831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:51:20.953131 12831 server_base.cc:1034] running on GCE node
W20260810 03:51:20.953166 12841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:20.953598 12846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:20.953645 12842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:20.954901 12831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:51:20.955775 12831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:51:20.956967 12831 hybrid_clock.cc:648] HybridClock initialized: now 1786333880956948 us; error 34 us; skew 500 ppm
I20260810 03:51:20.959065 12831 webserver.cc:460] Webserver started at http://127.27.153.66:44891/ using document root <none> and password file <none>
I20260810 03:51:20.959842 12831 fs_manager.cc:359] Metadata directory not provided
I20260810 03:51:20.959940 12831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:51:20.960197 12831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:51:20.963276 12831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "fe107091b189473184ef73455d4fd427"
format_stamp: "Formatted at 2026-08-10 03:51:20 on dist-test-slave-pww0"
server_key: "7c5aafd34639dbab05e66f80fe1bfdea"
server_key_iv: "b94b8ecf26c3d15bd9d976f2351f74c1"
server_key_version: "encryptionkey@0"
I20260810 03:51:20.963982 12831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "fe107091b189473184ef73455d4fd427"
format_stamp: "Formatted at 2026-08-10 03:51:20 on dist-test-slave-pww0"
server_key: "7c5aafd34639dbab05e66f80fe1bfdea"
server_key_iv: "b94b8ecf26c3d15bd9d976f2351f74c1"
server_key_version: "encryptionkey@0"
I20260810 03:51:20.968230 12831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:51:20.971235 12855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:20.972780 12831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:51:20.972922 12831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "fe107091b189473184ef73455d4fd427"
format_stamp: "Formatted at 2026-08-10 03:51:20 on dist-test-slave-pww0"
server_key: "7c5aafd34639dbab05e66f80fe1bfdea"
server_key_iv: "b94b8ecf26c3d15bd9d976f2351f74c1"
server_key_version: "encryptionkey@0"
I20260810 03:51:20.973058 12831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:51:20.994261 12831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:51:20.995039 12831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:51:20.995224 12831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:51:20.995900 12831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:51:20.996917 12831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:51:20.996965 12831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:20.997006 12831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:51:20.997026 12831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:21.006098 12831 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:44159
I20260810 03:51:21.006139 13050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:44159 every 8 connection(s)
I20260810 03:51:21.007124 12831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:51:21.008759 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 12831
I20260810 03:51:21.008878 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:51:21.009214 28261 external_mini_cluster.cc:1428] Setting key 567085f96c13f1812fcc45aad431d7c0
I20260810 03:51:21.011497 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:42171
--builtin_ntp_servers=127.27.153.84:34027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 03:51:21.017324 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:21.009248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:50467 (local address 127.27.153.126:42171)
0810 03:51:21.009401 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:21.009405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:21.010676 (+ 1271us) server_negotiation.cc:407] Connection header received
0810 03:51:21.012296 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:21.012299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:21.012357 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:21.012430 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:21.013384 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:21.013909 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:21.015086 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:21.015385 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:21.015514 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:21.016748 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:51:21.016754 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:51:21.016757 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:51:21.016775 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:51:21.016800 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:51:21.016811 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:51:21.016888 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:21.017067 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 03:51:21.017147 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 03:51:21.018303 13056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:21.009776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42171 (local address 127.27.153.66:50467)
0810 03:51:21.010512 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:21.010561 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:21.012066 (+ 1505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:21.012609 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:21.012616 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:21.012714 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:21.013225 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:21.013239 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:21.014042 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:21.014045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:21.014945 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:21.014956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:21.015601 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:21.015605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:21.015781 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:21.016436 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:51:21.016438 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:21.016484 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:51:21.016559 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:51:21.016902 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:51:21.016908 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:51:21.016913 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 03:51:21.017052 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 03:51:21.017347 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":159,"threads_started":1}
I20260810 03:51:21.019603 13051 heartbeater.cc:347] Connected to a master server at 127.27.153.126:42171
I20260810 03:51:21.019883 13051 heartbeater.cc:464] Registering TS with master...
I20260810 03:51:21.020459 13051 heartbeater.cc:511] Master 127.27.153.126:42171 requested a full tablet report, sending...
I20260810 03:51:21.021688 12480 ts_manager.cc:194] Registered new tserver with Master: fe107091b189473184ef73455d4fd427 (127.27.153.66:44159)
I20260810 03:51:21.022298 12480 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.153.66:50467
W20260810 03:51:21.134546 13057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:51:21.134835 13057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:51:21.134891 13057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:51:21.139372 13057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:51:21.139490 13057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:51:21.139575 13057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:51:21.145097 13057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:34027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:42171
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:51:21.146335 13057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:51:21.147861 13057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:51:21.158535 13057 server_base.cc:1034] running on GCE node
W20260810 03:51:21.158484 13069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:21.158491 13066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:21.158520 13067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:21.159242 13057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:51:21.160005 13057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:51:21.161194 13057 hybrid_clock.cc:648] HybridClock initialized: now 1786333881161174 us; error 34 us; skew 500 ppm
I20260810 03:51:21.163189 13057 webserver.cc:460] Webserver started at http://127.27.153.67:44943/ using document root <none> and password file <none>
I20260810 03:51:21.163848 13057 fs_manager.cc:359] Metadata directory not provided
I20260810 03:51:21.163918 13057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:51:21.164124 13057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:51:21.166374 13057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "1bc76054630b49b6aa9aa12fa3f0e908"
format_stamp: "Formatted at 2026-08-10 03:51:21 on dist-test-slave-pww0"
server_key: "73c60d8e0e8eb56479bb71e7f71c32f6"
server_key_iv: "f3521fe3fcaa0eccab59bf720e9dbf60"
server_key_version: "encryptionkey@0"
I20260810 03:51:21.166977 13057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "1bc76054630b49b6aa9aa12fa3f0e908"
format_stamp: "Formatted at 2026-08-10 03:51:21 on dist-test-slave-pww0"
server_key: "73c60d8e0e8eb56479bb71e7f71c32f6"
server_key_iv: "f3521fe3fcaa0eccab59bf720e9dbf60"
server_key_version: "encryptionkey@0"
I20260810 03:51:21.171319 13057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 03:51:21.174316 13077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:21.175702 13057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:51:21.175851 13057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "1bc76054630b49b6aa9aa12fa3f0e908"
format_stamp: "Formatted at 2026-08-10 03:51:21 on dist-test-slave-pww0"
server_key: "73c60d8e0e8eb56479bb71e7f71c32f6"
server_key_iv: "f3521fe3fcaa0eccab59bf720e9dbf60"
server_key_version: "encryptionkey@0"
I20260810 03:51:21.175974 13057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:51:21.197870 13057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:51:21.198733 13057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:51:21.198945 13057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:51:21.199652 13057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:51:21.200731 13057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:51:21.200785 13057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:21.200843 13057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:51:21.200871 13057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:21.209892 13057 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:34673
I20260810 03:51:21.209934 13273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:34673 every 8 connection(s)
I20260810 03:51:21.210896 13057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:51:21.211760 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 13057
I20260810 03:51:21.211858 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:51:21.212201 28261 external_mini_cluster.cc:1428] Setting key 59ec27a424a49f4e53915bcddd3618dc
I20260810 03:51:21.221786 12549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:21.213466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:50313 (local address 127.27.153.126:42171)
0810 03:51:21.213623 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:21.213628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:21.214893 (+ 1265us) server_negotiation.cc:407] Connection header received
0810 03:51:21.216438 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:21.216442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:21.216507 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:21.216580 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:21.217554 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:21.218085 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:21.219344 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:21.219630 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:21.219729 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:21.221248 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:51:21.221252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:51:21.221254 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:51:21.221268 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:51:21.221289 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:51:21.221297 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:51:21.221358 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:21.221521 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 03:51:21.221595 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:51:21.222738 13280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:21.214007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42171 (local address 127.27.153.67:50313)
0810 03:51:21.214755 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:21.214801 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:21.216208 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:21.216775 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:21.216782 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:21.216876 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:21.217398 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:21.217411 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:21.218232 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:21.218235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:21.219201 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:21.219213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:21.219800 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:21.219805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:21.219977 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:21.220930 (+ 953us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:51:21.220932 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:21.220982 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:51:21.221058 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:51:21.221372 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:51:21.221383 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:51:21.221389 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 03:51:21.221508 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 03:51:21.221839 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":132,"threads_started":1}
I20260810 03:51:21.223980 13274 heartbeater.cc:347] Connected to a master server at 127.27.153.126:42171
I20260810 03:51:21.224262 13274 heartbeater.cc:464] Registering TS with master...
I20260810 03:51:21.224849 13274 heartbeater.cc:511] Master 127.27.153.126:42171 requested a full tablet report, sending...
I20260810 03:51:21.226014 12480 ts_manager.cc:194] Registered new tserver with Master: 1bc76054630b49b6aa9aa12fa3f0e908 (127.27.153.67:34673)
I20260810 03:51:21.226612 12480 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.153.67:50313
I20260810 03:51:21.226999 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 03:51:21.230188 12549 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:21.229185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56830 (local address 127.27.153.126:42171)
0810 03:51:21.229319 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:21.229324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:21.229537 (+ 213us) server_negotiation.cc:407] Connection header received
0810 03:51:21.229666 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:21.229669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:21.229715 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:21.229780 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:21.230060 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56830: BlockingRecv error: recv got EOF from 127.0.0.1:56830 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 03:51:21.230376 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 12593
I20260810 03:51:21.248733 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 12831
I20260810 03:51:21.256894 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 13057
I20260810 03:51:21.265884 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 12429
2026-08-10T03:51:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (991 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T03:51:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:51:21Z Disabled control of system clock
I20260810 03:51:21.309206 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:45335
--webserver_interface=127.27.153.126
--webserver_port=0
--builtin_ntp_servers=127.27.153.84:46783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:45335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 03:51:21.425819 13298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:51:21.426095 13298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:51:21.426151 13298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:51:21.430265 13298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:51:21.430365 13298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:51:21.430387 13298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:51:21.430409 13298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:51:21.430428 13298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:51:21.435283 13298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:46783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:45335
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:45335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:51:21.436470 13298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:51:21.437932 13298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:51:21.446626 13315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:21.446667 13298 server_base.cc:1034] running on GCE node
W20260810 03:51:21.446651 13312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:21.446626 13321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:21.447505 13298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:51:21.448670 13298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:51:21.449836 13298 hybrid_clock.cc:648] HybridClock initialized: now 1786333881449819 us; error 32 us; skew 500 ppm
I20260810 03:51:21.451808 13298 webserver.cc:460] Webserver started at http://127.27.153.126:37327/ using document root <none> and password file <none>
I20260810 03:51:21.452376 13298 fs_manager.cc:359] Metadata directory not provided
I20260810 03:51:21.452440 13298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:51:21.452646 13298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:51:21.454820 13298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "8fb7812de363424d89a4971adf7b246b"
format_stamp: "Formatted at 2026-08-10 03:51:21 on dist-test-slave-pww0"
server_key: "8c5e2a6ba3c505fa3e4610f0849b0199"
server_key_iv: "9125ce6ed0936c8f5503c48448205e3b"
server_key_version: "encryptionkey@0"
I20260810 03:51:21.455417 13298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "8fb7812de363424d89a4971adf7b246b"
format_stamp: "Formatted at 2026-08-10 03:51:21 on dist-test-slave-pww0"
server_key: "8c5e2a6ba3c505fa3e4610f0849b0199"
server_key_iv: "9125ce6ed0936c8f5503c48448205e3b"
server_key_version: "encryptionkey@0"
I20260810 03:51:21.459327 13298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:51:21.462038 13329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:21.463554 13298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:51:21.463689 13298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "8fb7812de363424d89a4971adf7b246b"
format_stamp: "Formatted at 2026-08-10 03:51:21 on dist-test-slave-pww0"
server_key: "8c5e2a6ba3c505fa3e4610f0849b0199"
server_key_iv: "9125ce6ed0936c8f5503c48448205e3b"
server_key_version: "encryptionkey@0"
I20260810 03:51:21.463812 13298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:51:21.468691 13298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:51:21.469309 13298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:51:21.469475 13298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:51:21.476639 13298 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:45335
I20260810 03:51:21.476650 13420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:45335 every 8 connection(s)
I20260810 03:51:21.477625 13298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:51:21.481366 13422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:51:21.485946 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 13298
I20260810 03:51:21.486045 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:51:21.486332 28261 external_mini_cluster.cc:1428] Setting key a674004189ef2fd0146c3adaaeb12bb3
I20260810 03:51:21.488180 13422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fb7812de363424d89a4971adf7b246b: Bootstrap starting.
I20260810 03:51:21.490697 13422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fb7812de363424d89a4971adf7b246b: Neither blocks nor log segments found. Creating new log.
I20260810 03:51:21.491626 13422 log.cc:824] T 00000000000000000000000000000000 P 8fb7812de363424d89a4971adf7b246b: Log is configured to *not* fsync() on all Append() calls
I20260810 03:51:21.492758 13427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:21.487893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38242 (local address 127.27.153.126:45335)
0810 03:51:21.488694 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:21.488712 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:21.488749 (+ 37us) server_negotiation.cc:407] Connection header received
0810 03:51:21.489443 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:21.489463 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:21.489849 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:21.490183 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:21.491096 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:51:21.491100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:51:21.491113 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:51:21.491123 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:51:21.491189 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:51:21.491207 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:51:21.491306 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:21.491444 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 03:51:21.491747 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":179,"threads_started":1}
I20260810 03:51:21.494117 13422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fb7812de363424d89a4971adf7b246b: No bootstrap required, opened a new log
I20260810 03:51:21.498062 13422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fb7812de363424d89a4971adf7b246b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb7812de363424d89a4971adf7b246b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45335 } }
I20260810 03:51:21.498318 13422 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fb7812de363424d89a4971adf7b246b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:51:21.498368 13422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fb7812de363424d89a4971adf7b246b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fb7812de363424d89a4971adf7b246b, State: Initialized, Role: FOLLOWER
I20260810 03:51:21.498826 13422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fb7812de363424d89a4971adf7b246b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb7812de363424d89a4971adf7b246b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45335 } }
I20260810 03:51:21.498965 13422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fb7812de363424d89a4971adf7b246b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:51:21.499019 13422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fb7812de363424d89a4971adf7b246b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:51:21.499110 13422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fb7812de363424d89a4971adf7b246b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:51:21.500394 13422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fb7812de363424d89a4971adf7b246b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb7812de363424d89a4971adf7b246b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45335 } }
I20260810 03:51:21.500777 13422 leader_election.cc:302] T 00000000000000000000000000000000 P 8fb7812de363424d89a4971adf7b246b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fb7812de363424d89a4971adf7b246b; no voters:
I20260810 03:51:21.501097 13422 leader_election.cc:288] T 00000000000000000000000000000000 P 8fb7812de363424d89a4971adf7b246b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:51:21.501221 13434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fb7812de363424d89a4971adf7b246b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:51:21.501477 13434 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fb7812de363424d89a4971adf7b246b [term 1 LEADER]: Becoming Leader. State: Replica: 8fb7812de363424d89a4971adf7b246b, State: Running, Role: LEADER
I20260810 03:51:21.501880 13434 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fb7812de363424d89a4971adf7b246b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb7812de363424d89a4971adf7b246b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45335 } }
I20260810 03:51:21.502439 13422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fb7812de363424d89a4971adf7b246b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:51:21.503568 13436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb7812de363424d89a4971adf7b246b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fb7812de363424d89a4971adf7b246b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb7812de363424d89a4971adf7b246b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45335 } } }
I20260810 03:51:21.503723 13436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb7812de363424d89a4971adf7b246b [sys.catalog]: This master's current role is: LEADER
I20260810 03:51:21.504029 13444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb7812de363424d89a4971adf7b246b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fb7812de363424d89a4971adf7b246b. Latest consensus state: current_term: 1 leader_uuid: "8fb7812de363424d89a4971adf7b246b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fb7812de363424d89a4971adf7b246b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45335 } } }
I20260810 03:51:21.504137 13444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb7812de363424d89a4971adf7b246b [sys.catalog]: This master's current role is: LEADER
I20260810 03:51:21.504139 13454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:51:21.507277 13454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:51:21.513315 13454 catalog_manager.cc:1347] Generated new cluster ID: aaf76070e6984e60934aa387d58aec2b
I20260810 03:51:21.513409 13454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:51:21.544643 13454 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:51:21.545099 13454 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:51:21.550087 13454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fb7812de363424d89a4971adf7b246b: Generated new TSK 0
I20260810 03:51:21.550896 13454 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:51:21.564639 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:45335
--builtin_ntp_servers=127.27.153.84:46783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 03:51:21.679486 13471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:51:21.679785 13471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:51:21.679843 13471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:51:21.683998 13471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:51:21.684068 13471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:51:21.684127 13471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:51:21.689021 13471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:46783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:45335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:51:21.690116 13471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:51:21.691529 13471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:51:21.701354 13487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:21.701334 13486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:21.701493 13471 server_base.cc:1034] running on GCE node
W20260810 03:51:21.701678 13492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:21.702169 13471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:51:21.702916 13471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:51:21.704116 13471 hybrid_clock.cc:648] HybridClock initialized: now 1786333881704079 us; error 43 us; skew 500 ppm
I20260810 03:51:21.706121 13471 webserver.cc:460] Webserver started at http://127.27.153.65:41387/ using document root <none> and password file <none>
I20260810 03:51:21.706769 13471 fs_manager.cc:359] Metadata directory not provided
I20260810 03:51:21.706846 13471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:51:21.707055 13471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:51:21.709273 13471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "7d05a636cfa84f378373486934789564"
format_stamp: "Formatted at 2026-08-10 03:51:21 on dist-test-slave-pww0"
server_key: "d431cb219a19f5381505036ee8d3fa6c"
server_key_iv: "19031adcd5d32b52bb5a1c19285928e7"
server_key_version: "encryptionkey@0"
I20260810 03:51:21.709877 13471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "7d05a636cfa84f378373486934789564"
format_stamp: "Formatted at 2026-08-10 03:51:21 on dist-test-slave-pww0"
server_key: "d431cb219a19f5381505036ee8d3fa6c"
server_key_iv: "19031adcd5d32b52bb5a1c19285928e7"
server_key_version: "encryptionkey@0"
I20260810 03:51:21.714012 13471 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260810 03:51:21.716881 13500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:21.718302 13471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:51:21.718451 13471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "7d05a636cfa84f378373486934789564"
format_stamp: "Formatted at 2026-08-10 03:51:21 on dist-test-slave-pww0"
server_key: "d431cb219a19f5381505036ee8d3fa6c"
server_key_iv: "19031adcd5d32b52bb5a1c19285928e7"
server_key_version: "encryptionkey@0"
I20260810 03:51:21.718576 13471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:51:21.723623 13471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:51:21.724291 13471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:51:21.724466 13471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:51:21.725039 13471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:51:21.726011 13471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:51:21.726059 13471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:21.726121 13471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:51:21.726150 13471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:21.734475 13471 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:42647
I20260810 03:51:21.734500 13694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:42647 every 8 connection(s)
I20260810 03:51:21.735406 13471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:51:21.741741 13427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:21.737363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:41739 (local address 127.27.153.126:45335)
0810 03:51:21.737555 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:21.737560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:21.738645 (+ 1085us) server_negotiation.cc:407] Connection header received
0810 03:51:21.740019 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:21.740022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:21.740167 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:21.740258 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:21.741341 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:51:21.741344 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:51:21.741346 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:51:21.741358 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:51:21.741380 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:51:21.741388 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:51:21.741474 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:21.741563 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 03:51:21.741621 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 03:51:21.742460 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 13471
I20260810 03:51:21.742560 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:51:21.742682 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:21.737766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45335 (local address 127.27.153.65:41739)
0810 03:51:21.738486 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:21.738529 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:21.739809 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:21.740432 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:21.740440 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:21.740535 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:21.741067 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:51:21.741069 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:21.741112 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:51:21.741191 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:51:21.741458 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:51:21.741463 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:51:21.741467 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 03:51:21.741594 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 03:51:21.741810 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":116,"threads_started":1}
I20260810 03:51:21.742925 28261 external_mini_cluster.cc:1428] Setting key fe1be10bb033df123f2f2944c2f9d046
I20260810 03:51:21.743999 13697 heartbeater.cc:347] Connected to a master server at 127.27.153.126:45335
I20260810 03:51:21.744246 13697 heartbeater.cc:464] Registering TS with master...
I20260810 03:51:21.744686 13697 heartbeater.cc:511] Master 127.27.153.126:45335 requested a full tablet report, sending...
I20260810 03:51:21.744951 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:45335
--builtin_ntp_servers=127.27.153.84:46783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 03:51:21.746214 13360 ts_manager.cc:194] Registered new tserver with Master: 7d05a636cfa84f378373486934789564 (127.27.153.65:42647)
W20260810 03:51:21.859714 13701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:51:21.859994 13701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:51:21.860035 13701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:51:21.864154 13701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:51:21.864223 13701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:51:21.864281 13701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:51:21.869170 13701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:46783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:45335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:51:21.870288 13701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:51:21.871778 13701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:51:21.881369 13707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:21.881382 13708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:21.881382 13712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:21.881438 13701 server_base.cc:1034] running on GCE node
I20260810 03:51:21.882045 13701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:51:21.882689 13701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:51:21.883869 13701 hybrid_clock.cc:648] HybridClock initialized: now 1786333881883848 us; error 31 us; skew 500 ppm
I20260810 03:51:21.885866 13701 webserver.cc:460] Webserver started at http://127.27.153.66:34497/ using document root <none> and password file <none>
I20260810 03:51:21.886457 13701 fs_manager.cc:359] Metadata directory not provided
I20260810 03:51:21.886523 13701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:51:21.886725 13701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:51:21.888886 13701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "6b99efbfded2404f8483468002012f24"
format_stamp: "Formatted at 2026-08-10 03:51:21 on dist-test-slave-pww0"
server_key: "4c05593fcd23336dfaa0357f0a2edbc6"
server_key_iv: "728afc4d8f831505152405577332f572"
server_key_version: "encryptionkey@0"
I20260810 03:51:21.889472 13701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "6b99efbfded2404f8483468002012f24"
format_stamp: "Formatted at 2026-08-10 03:51:21 on dist-test-slave-pww0"
server_key: "4c05593fcd23336dfaa0357f0a2edbc6"
server_key_iv: "728afc4d8f831505152405577332f572"
server_key_version: "encryptionkey@0"
I20260810 03:51:21.893424 13701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 03:51:21.896229 13727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:21.897586 13701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:51:21.897745 13701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "6b99efbfded2404f8483468002012f24"
format_stamp: "Formatted at 2026-08-10 03:51:21 on dist-test-slave-pww0"
server_key: "4c05593fcd23336dfaa0357f0a2edbc6"
server_key_iv: "728afc4d8f831505152405577332f572"
server_key_version: "encryptionkey@0"
I20260810 03:51:21.897866 13701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:51:21.902750 13701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:51:21.903421 13701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:51:21.903618 13701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:51:21.904193 13701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:51:21.905176 13701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:51:21.905225 13701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:21.905278 13701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:51:21.905309 13701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:21.913897 13701 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:41547
I20260810 03:51:21.913949 13921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:41547 every 8 connection(s)
I20260810 03:51:21.914845 13701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:51:21.921129 13427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:21.916733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:44075 (local address 127.27.153.126:45335)
0810 03:51:21.916872 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:21.916876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:21.918010 (+ 1134us) server_negotiation.cc:407] Connection header received
0810 03:51:21.919372 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:21.919375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:21.919415 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:21.919540 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:21.920734 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:51:21.920737 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:51:21.920740 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:51:21.920753 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:51:21.920773 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:51:21.920781 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:51:21.920870 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:21.920963 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 03:51:21.921003 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 03:51:21.922050 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:21.917145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45335 (local address 127.27.153.66:44075)
0810 03:51:21.917853 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:21.917895 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:21.919071 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:21.919805 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:21.919813 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:21.919910 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:21.920455 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:51:21.920457 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:21.920500 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:51:21.920572 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:51:21.920855 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:51:21.920860 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:51:21.920864 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 03:51:21.920995 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 03:51:21.921209 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":109,"threads_started":1}
I20260810 03:51:21.922588 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 13701
I20260810 03:51:21.922705 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:51:21.923080 28261 external_mini_cluster.cc:1428] Setting key 662f7315e7091947d08a1f552004f1ec
I20260810 03:51:21.923497 13922 heartbeater.cc:347] Connected to a master server at 127.27.153.126:45335
I20260810 03:51:21.923769 13922 heartbeater.cc:464] Registering TS with master...
I20260810 03:51:21.924259 13922 heartbeater.cc:511] Master 127.27.153.126:45335 requested a full tablet report, sending...
I20260810 03:51:21.925262 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:45335
--builtin_ntp_servers=127.27.153.84:46783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 03:51:21.925381 13360 ts_manager.cc:194] Registered new tserver with Master: 6b99efbfded2404f8483468002012f24 (127.27.153.66:41547)
W20260810 03:51:22.043665 13926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:51:22.043962 13926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:51:22.044023 13926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:51:22.048326 13926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:51:22.048409 13926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:51:22.048482 13926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:51:22.053714 13926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:46783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:45335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:51:22.054824 13926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:51:22.056320 13926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:51:22.066371 13926 server_base.cc:1034] running on GCE node
W20260810 03:51:22.066316 13933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:22.066402 13934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:22.066316 13936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:22.067056 13926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:51:22.067781 13926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:51:22.068957 13926 hybrid_clock.cc:648] HybridClock initialized: now 1786333882068934 us; error 36 us; skew 500 ppm
I20260810 03:51:22.070888 13926 webserver.cc:460] Webserver started at http://127.27.153.67:44545/ using document root <none> and password file <none>
I20260810 03:51:22.071523 13926 fs_manager.cc:359] Metadata directory not provided
I20260810 03:51:22.071600 13926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:51:22.071810 13926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:51:22.073961 13926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "d214c250bc1f4202a3fab85e0330e98d"
format_stamp: "Formatted at 2026-08-10 03:51:22 on dist-test-slave-pww0"
server_key: "8e7eb5f05e6bb93069eb4e9dddc8eda2"
server_key_iv: "0614c6aab09523544f2f9a157599bacc"
server_key_version: "encryptionkey@0"
I20260810 03:51:22.074533 13926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "d214c250bc1f4202a3fab85e0330e98d"
format_stamp: "Formatted at 2026-08-10 03:51:22 on dist-test-slave-pww0"
server_key: "8e7eb5f05e6bb93069eb4e9dddc8eda2"
server_key_iv: "0614c6aab09523544f2f9a157599bacc"
server_key_version: "encryptionkey@0"
I20260810 03:51:22.078343 13926 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 03:51:22.081022 13948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:22.082345 13926 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 03:51:22.082453 13926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "d214c250bc1f4202a3fab85e0330e98d"
format_stamp: "Formatted at 2026-08-10 03:51:22 on dist-test-slave-pww0"
server_key: "8e7eb5f05e6bb93069eb4e9dddc8eda2"
server_key_iv: "0614c6aab09523544f2f9a157599bacc"
server_key_version: "encryptionkey@0"
I20260810 03:51:22.082572 13926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:51:22.087277 13926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:51:22.087961 13926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:51:22.088140 13926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:51:22.088716 13926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:51:22.089677 13926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:51:22.089723 13926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:22.089764 13926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:51:22.089792 13926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:22.098619 13926 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:45903
I20260810 03:51:22.098668 14148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:45903 every 8 connection(s)
I20260810 03:51:22.099642 13926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:51:22.103365 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 13926
I20260810 03:51:22.103483 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:51:22.103794 28261 external_mini_cluster.cc:1428] Setting key a4549fda7441931a43c164b7f7e2c788
I20260810 03:51:22.106174 13427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:22.101542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:59535 (local address 127.27.153.126:45335)
0810 03:51:22.101685 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:22.101689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:22.102944 (+ 1255us) server_negotiation.cc:407] Connection header received
0810 03:51:22.104438 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:22.104442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:22.104485 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:22.104553 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:22.105705 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:51:22.105709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:51:22.105712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:51:22.105725 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:51:22.105748 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:51:22.105759 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:51:22.105841 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:22.106022 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 03:51:22.106066 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 03:51:22.107116 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:22.101968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45335 (local address 127.27.153.67:59535)
0810 03:51:22.102762 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:22.102841 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:22.104239 (+ 1398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:22.104763 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:22.104771 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:22.104866 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:22.105439 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:51:22.105441 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:22.105484 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:51:22.105558 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:51:22.105855 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:51:22.105865 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:51:22.105871 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 03:51:22.106043 (+ 172us) client_negotiation.cc:240] Negotiation successful
0810 03:51:22.106285 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":140,"threads_started":1}
I20260810 03:51:22.108233 14150 heartbeater.cc:347] Connected to a master server at 127.27.153.126:45335
I20260810 03:51:22.108467 14150 heartbeater.cc:464] Registering TS with master...
I20260810 03:51:22.108902 14150 heartbeater.cc:511] Master 127.27.153.126:45335 requested a full tablet report, sending...
I20260810 03:51:22.109915 13360 ts_manager.cc:194] Registered new tserver with Master: d214c250bc1f4202a3fab85e0330e98d (127.27.153.67:45903)
I20260810 03:51:22.118314 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 03:51:22.121625 13427 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:51:22.120600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43614 (local address 127.27.153.126:45335)
0810 03:51:22.120761 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:22.120765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:22.120896 (+ 131us) server_negotiation.cc:407] Connection header received
0810 03:51:22.121046 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:22.121048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:22.121085 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:22.121144 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:22.121485 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43614: BlockingRecv error: recv got EOF from 127.0.0.1:43614 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 03:51:22.121768 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 13471
I20260810 03:51:22.137264 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 13701
I20260810 03:51:22.166706 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 13926
I20260810 03:51:22.189039 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 13298
2026-08-10T03:51:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (922 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:51:22 dist-test-slave-pww0 krb5kdc[14169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:51:22 dist-test-slave-pww0 krb5kdc[14169](info): set up 2 sockets
Aug 10 03:51:22 dist-test-slave-pww0 krb5kdc[14169](info): commencing operation
krb5kdc: starting...
W20260810 03:51:23.403916 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:51:23 dist-test-slave-pww0 krb5kdc[14169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:51:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:51:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:51:23.535730 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:43175
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:43175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:51:23.647861 14189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:51:23.648140 14189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:51:23.648182 14189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:51:23.652288 14189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:51:23.652350 14189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:51:23.652369 14189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:51:23.652384 14189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:51:23.652397 14189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:51:23.657769 14189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:43175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:43175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:51:23.658881 14189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:51:23.659967 14189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:51:23.668430 14199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:23.668419 14200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:23.668536 14189 server_base.cc:1034] running on GCE node
W20260810 03:51:23.668431 14203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:23.669237 14189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:51:23.670346 14189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:51:23.671550 14189 hybrid_clock.cc:648] HybridClock initialized: now 1786333883671524 us; error 39 us; skew 500 ppm
Aug 10 03:51:23 dist-test-slave-pww0 krb5kdc[14169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333883, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:51:23.674288 14189 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:51:23.675216 14189 webserver.cc:460] Webserver started at http://127.27.153.126:45469/ using document root <none> and password file <none>
I20260810 03:51:23.675814 14189 fs_manager.cc:359] Metadata directory not provided
I20260810 03:51:23.675879 14189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:51:23.676048 14189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:51:23.678071 14189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "5fa1ed16d9f34fd3a2025f2223ee0e3e"
format_stamp: "Formatted at 2026-08-10 03:51:23 on dist-test-slave-pww0"
server_key: "61e695b2d0ddb31bbe011376cf4d0d94"
server_key_iv: "a0eb835a7acc35dd5c1bfd2d86842770"
server_key_version: "encryptionkey@0"
I20260810 03:51:23.678578 14189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "5fa1ed16d9f34fd3a2025f2223ee0e3e"
format_stamp: "Formatted at 2026-08-10 03:51:23 on dist-test-slave-pww0"
server_key: "61e695b2d0ddb31bbe011376cf4d0d94"
server_key_iv: "a0eb835a7acc35dd5c1bfd2d86842770"
server_key_version: "encryptionkey@0"
I20260810 03:51:23.682350 14189 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 03:51:23.684934 14221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:23.686220 14189 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 03:51:23.686333 14189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "5fa1ed16d9f34fd3a2025f2223ee0e3e"
format_stamp: "Formatted at 2026-08-10 03:51:23 on dist-test-slave-pww0"
server_key: "61e695b2d0ddb31bbe011376cf4d0d94"
server_key_iv: "a0eb835a7acc35dd5c1bfd2d86842770"
server_key_version: "encryptionkey@0"
I20260810 03:51:23.686437 14189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:51:23.705817 14189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:51:23.709049 14189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:51:23.709213 14189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:51:23.715911 14189 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:43175
I20260810 03:51:23.716177 14309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:43175 every 8 connection(s)
I20260810 03:51:23.716898 14189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:51:23.719647 14311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:51:23.723524 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 14189
I20260810 03:51:23.723654 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:51:23.723969 28261 external_mini_cluster.cc:1428] Setting key 4bccbf98faf79931942b395ce56727be
I20260810 03:51:23.725908 14311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e: Bootstrap starting.
I20260810 03:51:23.728255 14311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e: Neither blocks nor log segments found. Creating new log.
I20260810 03:51:23.728987 14311 log.cc:824] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e: Log is configured to *not* fsync() on all Append() calls
I20260810 03:51:23.730878 14311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e: No bootstrap required, opened a new log
Aug 10 03:51:23 dist-test-slave-pww0 krb5kdc[14169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:51:23.734576 14311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa1ed16d9f34fd3a2025f2223ee0e3e" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 43175 } }
I20260810 03:51:23.734817 14311 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:51:23.734879 14311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fa1ed16d9f34fd3a2025f2223ee0e3e, State: Initialized, Role: FOLLOWER
I20260810 03:51:23.735320 14311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa1ed16d9f34fd3a2025f2223ee0e3e" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 43175 } }
I20260810 03:51:23.735471 14311 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:51:23.735541 14311 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:51:23.735668 14311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:51:23.736536 14311 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa1ed16d9f34fd3a2025f2223ee0e3e" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 43175 } }
I20260810 03:51:23.736876 14311 leader_election.cc:302] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fa1ed16d9f34fd3a2025f2223ee0e3e; no voters:
I20260810 03:51:23.737221 14311 leader_election.cc:288] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:51:23.737303 14318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:51:23.737505 14318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e [term 1 LEADER]: Becoming Leader. State: Replica: 5fa1ed16d9f34fd3a2025f2223ee0e3e, State: Running, Role: LEADER
I20260810 03:51:23.737814 14318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa1ed16d9f34fd3a2025f2223ee0e3e" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 43175 } }
I20260810 03:51:23.738011 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:23.725623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49864 (local address 127.27.153.126:43175)
0810 03:51:23.726124 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:23.726154 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:23.726191 (+ 37us) server_negotiation.cc:407] Connection header received
0810 03:51:23.726856 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:23.726875 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:23.727206 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:23.727553 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:23.728383 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:23.729412 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:23.730245 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:23.730730 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:23.730877 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:23.733312 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:51:23.733318 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:51:23.733331 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:51:23.733359 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:51:23.735012 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:51:23.735427 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:51:23.735432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:51:23.735436 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:51:23.735614 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:51:23.735897 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:51:23.735901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:51:23.735903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:51:23.736297 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:51:23.736474 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:23.736781 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 03:51:23.737094 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":166,"threads_started":1}
I20260810 03:51:23.738523 14311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:51:23.739037 14318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fa1ed16d9f34fd3a2025f2223ee0e3e. Latest consensus state: current_term: 1 leader_uuid: "5fa1ed16d9f34fd3a2025f2223ee0e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa1ed16d9f34fd3a2025f2223ee0e3e" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 43175 } } }
I20260810 03:51:23.739156 14318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e [sys.catalog]: This master's current role is: LEADER
I20260810 03:51:23.739558 14319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fa1ed16d9f34fd3a2025f2223ee0e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fa1ed16d9f34fd3a2025f2223ee0e3e" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 43175 } } }
I20260810 03:51:23.739656 14319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e [sys.catalog]: This master's current role is: LEADER
W20260810 03:51:23.743120 14338 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:51:23.743201 14338 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:51:23.743294 14324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:51:23.744237 14324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:51:23.749894 14324 catalog_manager.cc:1347] Generated new cluster ID: 1cbce559b60d48bebb6d3ad69ff4db82
I20260810 03:51:23.749967 14324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:51:23.765913 14324 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:51:23.767256 14324 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:51:23.780949 14324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e: Generated new TSK 0
I20260810 03:51:23.781766 14324 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.153.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
I20260810 03:51:23.818466 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:43175
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:51:23.948444 14352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:51:23.948717 14352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:51:23.948760 14352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:51:23.952883 14352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:51:23.952948 14352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:51:23.953006 14352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:51:23.958225 14352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:43175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:51:23.959281 14352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:51:23.960340 14352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:51:23.970506 14371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:23.970511 14362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:23.970569 14352 server_base.cc:1034] running on GCE node
W20260810 03:51:23.970506 14364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:23.971242 14352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:51:23.972196 14352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:51:23.973390 14352 hybrid_clock.cc:648] HybridClock initialized: now 1786333883973368 us; error 37 us; skew 500 ppm
Aug 10 03:51:23 dist-test-slave-pww0 krb5kdc[14169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333883, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:51:23.976341 14352 init.cc:377] Logged in from keytab as kudu/127.27.153.65@KRBTEST.COM (short username kudu)
I20260810 03:51:23.977252 14352 webserver.cc:460] Webserver started at http://127.27.153.65:41097/ using document root <none> and password file <none>
I20260810 03:51:23.977807 14352 fs_manager.cc:359] Metadata directory not provided
I20260810 03:51:23.977869 14352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:51:23.978056 14352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:51:23.980224 14352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "ca4253aefefc40a98f9c3d9309973b5c"
format_stamp: "Formatted at 2026-08-10 03:51:23 on dist-test-slave-pww0"
server_key: "42952d2a71bc12512122a83c20c47cef"
server_key_iv: "715bf4718489aeb1b3a3e2a5fc26de02"
server_key_version: "encryptionkey@0"
I20260810 03:51:23.980798 14352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "ca4253aefefc40a98f9c3d9309973b5c"
format_stamp: "Formatted at 2026-08-10 03:51:23 on dist-test-slave-pww0"
server_key: "42952d2a71bc12512122a83c20c47cef"
server_key_iv: "715bf4718489aeb1b3a3e2a5fc26de02"
server_key_version: "encryptionkey@0"
I20260810 03:51:23.984750 14352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 03:51:23.987426 14386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:23.988744 14352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:51:23.988873 14352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "ca4253aefefc40a98f9c3d9309973b5c"
format_stamp: "Formatted at 2026-08-10 03:51:23 on dist-test-slave-pww0"
server_key: "42952d2a71bc12512122a83c20c47cef"
server_key_iv: "715bf4718489aeb1b3a3e2a5fc26de02"
server_key_version: "encryptionkey@0"
I20260810 03:51:23.988979 14352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:51:24.011116 14352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:51:24.014322 14352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:51:24.014508 14352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:51:24.015112 14352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:51:24.016003 14352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:51:24.016052 14352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:24.016106 14352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:51:24.016132 14352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:24.024586 14352 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:46193
I20260810 03:51:24.024641 14583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:46193 every 8 connection(s)
I20260810 03:51:24.025470 14352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:51:24.029542 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 14352
I20260810 03:51:24.029659 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:51:24.029978 28261 external_mini_cluster.cc:1428] Setting key 68bf07005b96387b0b0882160aee56c5
Aug 10 03:51:24 dist-test-slave-pww0 krb5kdc[14169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333883, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:51:24.040934 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:24.027228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:57329 (local address 127.27.153.126:43175)
0810 03:51:24.027401 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:24.027405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:24.028312 (+ 907us) server_negotiation.cc:407] Connection header received
0810 03:51:24.029663 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:24.029668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:24.029728 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:24.029825 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:24.031125 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.031821 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.032951 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.033261 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.033370 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:24.036387 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:51:24.036392 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:51:24.036395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:51:24.036426 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:51:24.038088 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:51:24.038792 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:51:24.038796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:51:24.038798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:51:24.038902 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:51:24.039435 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:51:24.039439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:51:24.039441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:51:24.039739 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:51:24.039866 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:24.040599 (+ 733us) server_negotiation.cc:299] Negotiation successful
0810 03:51:24.040734 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
WARNING: no policy specified for kudu/127.27.153.66@KRBTEST.COM; defaulting to no policy
I20260810 03:51:24.042049 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:24.027663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:43175 (local address 127.27.153.65:57329)
0810 03:51:24.028144 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:24.028203 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:24.029429 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:24.029993 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:24.030000 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:24.030455 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:24.030991 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:24.031006 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.031943 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.031946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:24.032829 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:24.032839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.033454 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.033458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:24.033656 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:24.034297 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:24.034318 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:24.036186 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:51:24.038209 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:51:24.038216 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:51:24.038233 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:51:24.038644 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:51:24.039027 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:51:24.039030 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:51:24.039032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:51:24.039304 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:51:24.039931 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:51:24.039939 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:51:24.040301 (+ 362us) client_negotiation.cc:769] Sending connection context
0810 03:51:24.040546 (+ 245us) client_negotiation.cc:240] Negotiation successful
0810 03:51:24.040954 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":108,"threads_started":1}
I20260810 03:51:24.043510 14584 heartbeater.cc:347] Connected to a master server at 127.27.153.126:43175
I20260810 03:51:24.043812 14584 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.66@KRBTEST.COM" created.
I20260810 03:51:24.044392 14584 heartbeater.cc:511] Master 127.27.153.126:43175 requested a full tablet report, sending...
I20260810 03:51:24.046182 14248 ts_manager.cc:194] Registered new tserver with Master: ca4253aefefc40a98f9c3d9309973b5c (127.27.153.65:46193)
I20260810 03:51:24.047375 14248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:57329
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
I20260810 03:51:24.062798 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:43175
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:51:24.179910 14590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:51:24.180205 14590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:51:24.180249 14590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:51:24.184468 14590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:51:24.184547 14590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:51:24.184623 14590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:51:24.190524 14590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:43175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:51:24.191763 14590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:51:24.192919 14590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:51:24.203749 14603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:24.203863 14606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:24.203842 14590 server_base.cc:1034] running on GCE node
W20260810 03:51:24.203755 14604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:24.204435 14590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:51:24.205156 14590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:51:24.206351 14590 hybrid_clock.cc:648] HybridClock initialized: now 1786333884206330 us; error 35 us; skew 500 ppm
Aug 10 03:51:24 dist-test-slave-pww0 krb5kdc[14169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333884, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:51:24.212973 14590 init.cc:377] Logged in from keytab as kudu/127.27.153.66@KRBTEST.COM (short username kudu)
I20260810 03:51:24.214041 14590 webserver.cc:460] Webserver started at http://127.27.153.66:39163/ using document root <none> and password file <none>
I20260810 03:51:24.214639 14590 fs_manager.cc:359] Metadata directory not provided
I20260810 03:51:24.214716 14590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:51:24.214952 14590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:51:24.217197 14590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "26f1c3f1df7e4ad0b55c4c9f48dd88a5"
format_stamp: "Formatted at 2026-08-10 03:51:24 on dist-test-slave-pww0"
server_key: "72fe0648742ddfb0a4792f4fe8ee3f3f"
server_key_iv: "79fe076a1e8377f545833635f323d9ab"
server_key_version: "encryptionkey@0"
I20260810 03:51:24.217788 14590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "26f1c3f1df7e4ad0b55c4c9f48dd88a5"
format_stamp: "Formatted at 2026-08-10 03:51:24 on dist-test-slave-pww0"
server_key: "72fe0648742ddfb0a4792f4fe8ee3f3f"
server_key_iv: "79fe076a1e8377f545833635f323d9ab"
server_key_version: "encryptionkey@0"
I20260810 03:51:24.221792 14590 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:51:24.224583 14621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:24.225973 14590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 03:51:24.226104 14590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "26f1c3f1df7e4ad0b55c4c9f48dd88a5"
format_stamp: "Formatted at 2026-08-10 03:51:24 on dist-test-slave-pww0"
server_key: "72fe0648742ddfb0a4792f4fe8ee3f3f"
server_key_iv: "79fe076a1e8377f545833635f323d9ab"
server_key_version: "encryptionkey@0"
I20260810 03:51:24.226222 14590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:51:24.242784 14590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:51:24.246270 14590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:51:24.246495 14590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:51:24.247166 14590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:51:24.248188 14590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:51:24.248253 14590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:24.248313 14590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:51:24.248339 14590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:24.257507 14590 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:35725
I20260810 03:51:24.257540 14819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:35725 every 8 connection(s)
I20260810 03:51:24.258443 14590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:51:24.260919 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 14590
I20260810 03:51:24.261049 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:51:24.261392 28261 external_mini_cluster.cc:1428] Setting key 58d42c625e07f59a8e530565c2c41515
Aug 10 03:51:24 dist-test-slave-pww0 krb5kdc[14169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333884, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.153.67@KRBTEST.COM; defaulting to no policy
I20260810 03:51:24.273475 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:24.260379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:55035 (local address 127.27.153.126:43175)
0810 03:51:24.260545 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:24.260549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:24.261474 (+ 925us) server_negotiation.cc:407] Connection header received
0810 03:51:24.263178 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:24.263181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:24.263234 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:24.263308 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:24.264571 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.265329 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.266630 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.266948 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.267050 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:24.269716 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:51:24.269719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:51:24.269721 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:51:24.269746 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:51:24.271140 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:51:24.271794 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:51:24.271797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:51:24.271798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:51:24.271870 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:51:24.272293 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:51:24.272296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:51:24.272297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:51:24.272510 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:51:24.272612 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:24.273216 (+ 604us) server_negotiation.cc:299] Negotiation successful
0810 03:51:24.273341 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 03:51:24.274300 14822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:24.260769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:43175 (local address 127.27.153.66:55035)
0810 03:51:24.261317 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:24.261362 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:24.262957 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:24.263540 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:24.263548 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:24.263921 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:24.264428 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:24.264442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.265488 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.265493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:24.266506 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:24.266518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.267105 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.267109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:24.267279 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:24.267898 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:24.267915 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:24.269527 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:51:24.271266 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:51:24.271273 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:51:24.271291 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:51:24.271685 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:51:24.271962 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:51:24.271966 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:51:24.271968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:51:24.272153 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:51:24.272611 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:51:24.272616 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:51:24.272947 (+ 331us) client_negotiation.cc:769] Sending connection context
0810 03:51:24.273109 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 03:51:24.273384 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":132,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.67@KRBTEST.COM" created.
I20260810 03:51:24.275599 14820 heartbeater.cc:347] Connected to a master server at 127.27.153.126:43175
I20260810 03:51:24.275846 14820 heartbeater.cc:464] Registering TS with master...
I20260810 03:51:24.276360 14820 heartbeater.cc:511] Master 127.27.153.126:43175 requested a full tablet report, sending...
I20260810 03:51:24.277477 14248 ts_manager.cc:194] Registered new tserver with Master: 26f1c3f1df7e4ad0b55c4c9f48dd88a5 (127.27.153.66:35725)
I20260810 03:51:24.278033 14248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:55035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
I20260810 03:51:24.290961 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:43175
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:51:24.420837 14827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:51:24.421150 14827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:51:24.421195 14827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:51:24.425374 14827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:51:24.425452 14827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:51:24.425513 14827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:51:24.431141 14827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:43175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:51:24.432294 14827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:51:24.433305 14827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:51:24.443280 14838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:24.443297 14841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:51:24.443303 14836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:51:24.443534 14827 server_base.cc:1034] running on GCE node
I20260810 03:51:24.443948 14827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:51:24.444600 14827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:51:24.445783 14827 hybrid_clock.cc:648] HybridClock initialized: now 1786333884445762 us; error 35 us; skew 500 ppm
Aug 10 03:51:24 dist-test-slave-pww0 krb5kdc[14169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333884, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:51:24.448686 14827 init.cc:377] Logged in from keytab as kudu/127.27.153.67@KRBTEST.COM (short username kudu)
I20260810 03:51:24.449581 14827 webserver.cc:460] Webserver started at http://127.27.153.67:33571/ using document root <none> and password file <none>
I20260810 03:51:24.450137 14827 fs_manager.cc:359] Metadata directory not provided
I20260810 03:51:24.450207 14827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:51:24.450421 14827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:51:24.452543 14827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "8f3d18e9ac344fd1ad737830758ac79d"
format_stamp: "Formatted at 2026-08-10 03:51:24 on dist-test-slave-pww0"
server_key: "f686820d542617a9d9b729792a3592a0"
server_key_iv: "49457c8e000db2c68baa36288d389210"
server_key_version: "encryptionkey@0"
I20260810 03:51:24.453128 14827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "8f3d18e9ac344fd1ad737830758ac79d"
format_stamp: "Formatted at 2026-08-10 03:51:24 on dist-test-slave-pww0"
server_key: "f686820d542617a9d9b729792a3592a0"
server_key_iv: "49457c8e000db2c68baa36288d389210"
server_key_version: "encryptionkey@0"
I20260810 03:51:24.456858 14827 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:51:24.459391 14855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:24.460853 14827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:51:24.460986 14827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "8f3d18e9ac344fd1ad737830758ac79d"
format_stamp: "Formatted at 2026-08-10 03:51:24 on dist-test-slave-pww0"
server_key: "f686820d542617a9d9b729792a3592a0"
server_key_iv: "49457c8e000db2c68baa36288d389210"
server_key_version: "encryptionkey@0"
I20260810 03:51:24.461097 14827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:51:24.488322 14827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:51:24.491714 14827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:51:24.491914 14827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:51:24.492550 14827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:51:24.493494 14827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:51:24.493543 14827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:24.493599 14827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:51:24.493629 14827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:51:24.502331 14827 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:38131
I20260810 03:51:24.502408 15049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:38131 every 8 connection(s)
I20260810 03:51:24.503293 14827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:51:24.511579 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 14827
I20260810 03:51:24.511732 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:51:24.512064 28261 external_mini_cluster.cc:1428] Setting key dcaca8277e0c3d83f39d0353001fb88a
Aug 10 03:51:24 dist-test-slave-pww0 krb5kdc[14169](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333884, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:51:24.517198 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:24.505301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:57655 (local address 127.27.153.126:43175)
0810 03:51:24.505486 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:24.505490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:24.506336 (+ 846us) server_negotiation.cc:407] Connection header received
0810 03:51:24.507761 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:24.507765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:24.507822 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:24.507898 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:24.509093 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.509626 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.510794 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.511011 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.511103 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:24.513971 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:51:24.513975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:51:24.513978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:51:24.514007 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:51:24.515218 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:51:24.515734 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:51:24.515737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:51:24.515738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:51:24.515815 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:51:24.516149 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:51:24.516152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:51:24.516153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:51:24.516387 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:51:24.516473 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:24.516949 (+ 476us) server_negotiation.cc:299] Negotiation successful
0810 03:51:24.517058 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 03:51:24.518031 15056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:24.505688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:43175 (local address 127.27.153.67:57655)
0810 03:51:24.506171 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:24.506216 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:24.507547 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:24.508085 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:24.508092 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:24.508463 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:51:24.508938 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:24.508951 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.509761 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.509764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:24.510664 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:24.510674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.511156 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.511168 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:24.511387 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:24.512054 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:51:24.512075 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:51:24.513809 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:51:24.515323 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:51:24.515328 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:51:24.515340 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:51:24.515655 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:51:24.515885 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:51:24.515887 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:51:24.515888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:51:24.516023 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:51:24.516490 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:51:24.516496 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:51:24.516747 (+ 251us) client_negotiation.cc:769] Sending connection context
0810 03:51:24.516899 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 03:51:24.517170 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":116,"threads_started":1}
I20260810 03:51:24.519165 15050 heartbeater.cc:347] Connected to a master server at 127.27.153.126:43175
I20260810 03:51:24.519388 15050 heartbeater.cc:464] Registering TS with master...
I20260810 03:51:24.519891 15050 heartbeater.cc:511] Master 127.27.153.126:43175 requested a full tablet report, sending...
I20260810 03:51:24.520922 14248 ts_manager.cc:194] Registered new tserver with Master: 8f3d18e9ac344fd1ad737830758ac79d (127.27.153.67:38131)
I20260810 03:51:24.521534 14248 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:57655
I20260810 03:51:24.526176 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:51:24.535501 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:24.528212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49870 (local address 127.27.153.126:43175)
0810 03:51:24.528358 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:24.528363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:24.528490 (+ 127us) server_negotiation.cc:407] Connection header received
0810 03:51:24.528596 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:24.528598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:24.528639 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:24.528701 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:51:24.529356 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.529933 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.530814 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.531053 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.531158 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:24.532468 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:51:24.532471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:51:24.532473 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:51:24.532498 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:51:24.533755 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:51:24.534135 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:51:24.534137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:51:24.534139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:51:24.534213 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:51:24.534433 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:51:24.534436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:51:24.534438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:51:24.534746 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:51:24.534879 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:24.535173 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 03:51:24.535310 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:51:24.538396 14248 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:51:24.541721 14248 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:51:24.556253 15077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:24.550316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:38131 (local address 127.0.0.1:51456)
0810 03:51:24.550681 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:24.550736 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:24.550868 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:24.552014 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:24.552021 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:24.552047 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:51:24.552223 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:24.552231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.553368 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.553371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:24.554786 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:24.554797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.555707 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.555710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:24.555819 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:24.555858 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 03:51:24.555964 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 03:51:24.556093 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":52,"thread_start_us":58,"threads_started":1}
I20260810 03:51:24.556259 15080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:24.550523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51456 (local address 127.27.153.67:38131)
0810 03:51:24.551492 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:24.551505 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:24.551532 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:51:24.551622 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:24.551627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:24.551837 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:24.552005 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:51:24.552351 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.553237 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.554925 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.555574 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.555680 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:24.555839 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:24.556006 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 03:51:24.556107 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":837,"thread_start_us":181,"threads_started":1}
I20260810 03:51:24.556609 15082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:24.550676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:46193 (local address 127.0.0.1:47480)
0810 03:51:24.551110 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:24.551123 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:24.551221 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:24.552170 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:24.552172 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:24.552189 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:51:24.552277 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:24.552283 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.553674 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.553677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:24.554842 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:24.554849 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.556123 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.556127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:24.556242 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:24.556285 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 03:51:24.556355 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 03:51:24.556443 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":37,"thread_start_us":117,"threads_started":1}
I20260810 03:51:24.556761 15086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:24.551147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47480 (local address 127.27.153.65:46193)
0810 03:51:24.551656 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:24.551661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:24.551681 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:51:24.551743 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:24.551747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:24.551886 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:24.552005 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:51:24.552410 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.553544 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.554957 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.555816 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.555917 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:24.556056 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:24.556528 (+ 472us) server_negotiation.cc:299] Negotiation successful
0810 03:51:24.556630 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":126,"threads_started":1}
I20260810 03:51:24.556923 15087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:24.551429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.27.153.66:35725)
0810 03:51:24.551880 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:24.551888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:24.552207 (+ 319us) server_negotiation.cc:407] Connection header received
0810 03:51:24.552292 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:24.552297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:24.552501 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:24.552654 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:51:24.552855 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.553863 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.555021 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.555811 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.555918 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:24.556056 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:24.556674 (+ 618us) server_negotiation.cc:299] Negotiation successful
0810 03:51:24.556768 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":122,"threads_started":1}
I20260810 03:51:24.556613 15084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:24.551011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:35725 (local address 127.0.0.1:34480)
0810 03:51:24.552019 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:24.552032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:24.552114 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:24.552649 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:24.552652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:24.552672 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:51:24.552757 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:24.552762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.553996 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.553999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:24.554893 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:24.554906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.556123 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.556127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:24.556277 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:24.556294 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 03:51:24.556355 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 03:51:24.556443 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"mutex_wait_us":47,"thread_start_us":102,"threads_started":1}
I20260810 03:51:24.559391 14472 tablet_service.cc:1467] Processing CreateTablet for tablet e3044319fce04ee597334b74efd18a45 (DEFAULT_TABLE table=test-table [id=06e6f4f92d6244c8badb13f01d0bdf9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:51:24.559394 14940 tablet_service.cc:1467] Processing CreateTablet for tablet e3044319fce04ee597334b74efd18a45 (DEFAULT_TABLE table=test-table [id=06e6f4f92d6244c8badb13f01d0bdf9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:51:24.560365 14472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3044319fce04ee597334b74efd18a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:51:24.560462 14940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3044319fce04ee597334b74efd18a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:51:24.561641 14701 tablet_service.cc:1467] Processing CreateTablet for tablet e3044319fce04ee597334b74efd18a45 (DEFAULT_TABLE table=test-table [id=06e6f4f92d6244c8badb13f01d0bdf9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:51:24.562759 14701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3044319fce04ee597334b74efd18a45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:51:24.567013 15088 tablet_bootstrap.cc:492] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c: Bootstrap starting.
I20260810 03:51:24.568459 15091 tablet_bootstrap.cc:492] T e3044319fce04ee597334b74efd18a45 P 8f3d18e9ac344fd1ad737830758ac79d: Bootstrap starting.
I20260810 03:51:24.569548 15088 tablet_bootstrap.cc:654] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c: Neither blocks nor log segments found. Creating new log.
I20260810 03:51:24.570477 15088 log.cc:824] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c: Log is configured to *not* fsync() on all Append() calls
I20260810 03:51:24.570653 15091 tablet_bootstrap.cc:654] T e3044319fce04ee597334b74efd18a45 P 8f3d18e9ac344fd1ad737830758ac79d: Neither blocks nor log segments found. Creating new log.
I20260810 03:51:24.570914 15092 tablet_bootstrap.cc:492] T e3044319fce04ee597334b74efd18a45 P 26f1c3f1df7e4ad0b55c4c9f48dd88a5: Bootstrap starting.
I20260810 03:51:24.571991 15091 log.cc:824] T e3044319fce04ee597334b74efd18a45 P 8f3d18e9ac344fd1ad737830758ac79d: Log is configured to *not* fsync() on all Append() calls
I20260810 03:51:24.572705 15088 tablet_bootstrap.cc:492] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c: No bootstrap required, opened a new log
I20260810 03:51:24.572906 15088 ts_tablet_manager.cc:1397] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:51:24.573349 15092 tablet_bootstrap.cc:654] T e3044319fce04ee597334b74efd18a45 P 26f1c3f1df7e4ad0b55c4c9f48dd88a5: Neither blocks nor log segments found. Creating new log.
I20260810 03:51:24.574008 15092 log.cc:824] T e3044319fce04ee597334b74efd18a45 P 26f1c3f1df7e4ad0b55c4c9f48dd88a5: Log is configured to *not* fsync() on all Append() calls
I20260810 03:51:24.574116 15091 tablet_bootstrap.cc:492] T e3044319fce04ee597334b74efd18a45 P 8f3d18e9ac344fd1ad737830758ac79d: No bootstrap required, opened a new log
I20260810 03:51:24.574307 15091 ts_tablet_manager.cc:1397] T e3044319fce04ee597334b74efd18a45 P 8f3d18e9ac344fd1ad737830758ac79d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:51:24.576241 15092 tablet_bootstrap.cc:492] T e3044319fce04ee597334b74efd18a45 P 26f1c3f1df7e4ad0b55c4c9f48dd88a5: No bootstrap required, opened a new log
I20260810 03:51:24.576424 15092 ts_tablet_manager.cc:1397] T e3044319fce04ee597334b74efd18a45 P 26f1c3f1df7e4ad0b55c4c9f48dd88a5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:51:24.577462 15088 raft_consensus.cc:348] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3d18e9ac344fd1ad737830758ac79d" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 38131 } } peers { permanent_uuid: "26f1c3f1df7e4ad0b55c4c9f48dd88a5" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 35725 } } peers { permanent_uuid: "ca4253aefefc40a98f9c3d9309973b5c" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 46193 } }
I20260810 03:51:24.577755 15088 raft_consensus.cc:374] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:51:24.577816 15088 raft_consensus.cc:729] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca4253aefefc40a98f9c3d9309973b5c, State: Initialized, Role: FOLLOWER
I20260810 03:51:24.578092 15091 raft_consensus.cc:348] T e3044319fce04ee597334b74efd18a45 P 8f3d18e9ac344fd1ad737830758ac79d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3d18e9ac344fd1ad737830758ac79d" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 38131 } } peers { permanent_uuid: "26f1c3f1df7e4ad0b55c4c9f48dd88a5" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 35725 } } peers { permanent_uuid: "ca4253aefefc40a98f9c3d9309973b5c" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 46193 } }
I20260810 03:51:24.578337 15091 raft_consensus.cc:374] T e3044319fce04ee597334b74efd18a45 P 8f3d18e9ac344fd1ad737830758ac79d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:51:24.578392 15091 raft_consensus.cc:729] T e3044319fce04ee597334b74efd18a45 P 8f3d18e9ac344fd1ad737830758ac79d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f3d18e9ac344fd1ad737830758ac79d, State: Initialized, Role: FOLLOWER
I20260810 03:51:24.578333 15088 consensus_queue.cc:260] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3d18e9ac344fd1ad737830758ac79d" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 38131 } } peers { permanent_uuid: "26f1c3f1df7e4ad0b55c4c9f48dd88a5" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 35725 } } peers { permanent_uuid: "ca4253aefefc40a98f9c3d9309973b5c" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 46193 } }
I20260810 03:51:24.578825 15091 consensus_queue.cc:260] T e3044319fce04ee597334b74efd18a45 P 8f3d18e9ac344fd1ad737830758ac79d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3d18e9ac344fd1ad737830758ac79d" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 38131 } } peers { permanent_uuid: "26f1c3f1df7e4ad0b55c4c9f48dd88a5" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 35725 } } peers { permanent_uuid: "ca4253aefefc40a98f9c3d9309973b5c" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 46193 } }
I20260810 03:51:24.579402 15088 ts_tablet_manager.cc:1428] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 03:51:24.579563 15050 heartbeater.cc:503] Master 127.27.153.126:43175 was elected leader, sending a full tablet report...
I20260810 03:51:24.579744 14584 heartbeater.cc:503] Master 127.27.153.126:43175 was elected leader, sending a full tablet report...
I20260810 03:51:24.579622 15092 raft_consensus.cc:348] T e3044319fce04ee597334b74efd18a45 P 26f1c3f1df7e4ad0b55c4c9f48dd88a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3d18e9ac344fd1ad737830758ac79d" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 38131 } } peers { permanent_uuid: "26f1c3f1df7e4ad0b55c4c9f48dd88a5" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 35725 } } peers { permanent_uuid: "ca4253aefefc40a98f9c3d9309973b5c" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 46193 } }
I20260810 03:51:24.579842 15092 raft_consensus.cc:374] T e3044319fce04ee597334b74efd18a45 P 26f1c3f1df7e4ad0b55c4c9f48dd88a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:51:24.579900 15092 raft_consensus.cc:729] T e3044319fce04ee597334b74efd18a45 P 26f1c3f1df7e4ad0b55c4c9f48dd88a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f1c3f1df7e4ad0b55c4c9f48dd88a5, State: Initialized, Role: FOLLOWER
I20260810 03:51:24.580281 15092 consensus_queue.cc:260] T e3044319fce04ee597334b74efd18a45 P 26f1c3f1df7e4ad0b55c4c9f48dd88a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3d18e9ac344fd1ad737830758ac79d" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 38131 } } peers { permanent_uuid: "26f1c3f1df7e4ad0b55c4c9f48dd88a5" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 35725 } } peers { permanent_uuid: "ca4253aefefc40a98f9c3d9309973b5c" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 46193 } }
I20260810 03:51:24.580946 14820 heartbeater.cc:503] Master 127.27.153.126:43175 was elected leader, sending a full tablet report...
I20260810 03:51:24.582165 15091 ts_tablet_manager.cc:1428] T e3044319fce04ee597334b74efd18a45 P 8f3d18e9ac344fd1ad737830758ac79d: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260810 03:51:24.582641 15092 ts_tablet_manager.cc:1428] T e3044319fce04ee597334b74efd18a45 P 26f1c3f1df7e4ad0b55c4c9f48dd88a5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:51:24.655745 15101 raft_consensus.cc:482] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:51:24.655994 15101 raft_consensus.cc:504] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3d18e9ac344fd1ad737830758ac79d" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 38131 } } peers { permanent_uuid: "26f1c3f1df7e4ad0b55c4c9f48dd88a5" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 35725 } } peers { permanent_uuid: "ca4253aefefc40a98f9c3d9309973b5c" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 46193 } }
I20260810 03:51:24.657184 15101 leader_election.cc:288] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f3d18e9ac344fd1ad737830758ac79d (127.27.153.67:38131), 26f1c3f1df7e4ad0b55c4c9f48dd88a5 (127.27.153.66:35725)
I20260810 03:51:24.661283 15080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:24.657599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:50697 (local address 127.27.153.67:38131)
0810 03:51:24.657788 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:24.657793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:24.657892 (+ 99us) server_negotiation.cc:407] Connection header received
0810 03:51:24.658040 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:24.658043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:24.658101 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:24.658180 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:51:24.658509 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.659116 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.660315 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.660741 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.660845 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:24.660900 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:24.661084 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 03:51:24.661158 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 03:51:24.661290 15087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:24.658030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:58497 (local address 127.27.153.66:35725)
0810 03:51:24.658182 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:24.658187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:24.658270 (+ 83us) server_negotiation.cc:407] Connection header received
0810 03:51:24.658409 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:24.658412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:24.658466 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:24.658536 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:51:24.658788 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.659315 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.660372 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.660765 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.660872 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:24.660923 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:24.661103 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 03:51:24.661187 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 03:51:24.661315 15112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:24.657880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:35725 (local address 127.27.153.65:58497)
0810 03:51:24.658185 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:24.658198 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:24.658310 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:24.658556 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:24.658558 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:24.658576 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:51:24.658683 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:24.658690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.659419 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.659422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:24.660235 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:24.660247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.660879 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.660883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:24.660984 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:24.661001 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 03:51:24.661072 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 03:51:24.661143 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"spinlock_wait_cycles":1664,"thread_start_us":101,"threads_started":1}
I20260810 03:51:24.661319 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:24.657446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:38131 (local address 127.27.153.65:50697)
0810 03:51:24.657788 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:24.657804 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:51:24.657935 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:51:24.658204 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:51:24.658207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:51:24.658232 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:51:24.658388 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:24.658395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.659234 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.659238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:24.660180 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:51:24.660191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.660865 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.660869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:51:24.660960 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:24.660983 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 03:51:24.661042 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 03:51:24.661113 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":1,"spinlock_wait_cycles":3968,"thread_start_us":141,"threads_started":1}
I20260810 03:51:24.662101 14973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3044319fce04ee597334b74efd18a45" candidate_uuid: "ca4253aefefc40a98f9c3d9309973b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f3d18e9ac344fd1ad737830758ac79d" is_pre_election: true
I20260810 03:51:24.662101 14742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3044319fce04ee597334b74efd18a45" candidate_uuid: "ca4253aefefc40a98f9c3d9309973b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1c3f1df7e4ad0b55c4c9f48dd88a5" is_pre_election: true
I20260810 03:51:24.662425 14742 raft_consensus.cc:2454] T e3044319fce04ee597334b74efd18a45 P 26f1c3f1df7e4ad0b55c4c9f48dd88a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca4253aefefc40a98f9c3d9309973b5c in term 0.
I20260810 03:51:24.662431 14973 raft_consensus.cc:2454] T e3044319fce04ee597334b74efd18a45 P 8f3d18e9ac344fd1ad737830758ac79d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca4253aefefc40a98f9c3d9309973b5c in term 0.
I20260810 03:51:24.662873 14393 leader_election.cc:302] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1c3f1df7e4ad0b55c4c9f48dd88a5, ca4253aefefc40a98f9c3d9309973b5c; no voters:
I20260810 03:51:24.663133 15101 raft_consensus.cc:2781] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:51:24.663228 15101 raft_consensus.cc:482] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:51:24.663280 15101 raft_consensus.cc:3037] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:51:24.664379 15101 raft_consensus.cc:504] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3d18e9ac344fd1ad737830758ac79d" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 38131 } } peers { permanent_uuid: "26f1c3f1df7e4ad0b55c4c9f48dd88a5" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 35725 } } peers { permanent_uuid: "ca4253aefefc40a98f9c3d9309973b5c" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 46193 } }
I20260810 03:51:24.664722 15101 leader_election.cc:288] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c [CANDIDATE]: Term 1 election: Requested vote from peers 8f3d18e9ac344fd1ad737830758ac79d (127.27.153.67:38131), 26f1c3f1df7e4ad0b55c4c9f48dd88a5 (127.27.153.66:35725)
I20260810 03:51:24.665191 14973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3044319fce04ee597334b74efd18a45" candidate_uuid: "ca4253aefefc40a98f9c3d9309973b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f3d18e9ac344fd1ad737830758ac79d"
I20260810 03:51:24.665194 14742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3044319fce04ee597334b74efd18a45" candidate_uuid: "ca4253aefefc40a98f9c3d9309973b5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f1c3f1df7e4ad0b55c4c9f48dd88a5"
I20260810 03:51:24.665313 14973 raft_consensus.cc:3037] T e3044319fce04ee597334b74efd18a45 P 8f3d18e9ac344fd1ad737830758ac79d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:51:24.665352 14742 raft_consensus.cc:3037] T e3044319fce04ee597334b74efd18a45 P 26f1c3f1df7e4ad0b55c4c9f48dd88a5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:51:24.666543 14742 raft_consensus.cc:2454] T e3044319fce04ee597334b74efd18a45 P 26f1c3f1df7e4ad0b55c4c9f48dd88a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca4253aefefc40a98f9c3d9309973b5c in term 1.
I20260810 03:51:24.666545 14973 raft_consensus.cc:2454] T e3044319fce04ee597334b74efd18a45 P 8f3d18e9ac344fd1ad737830758ac79d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca4253aefefc40a98f9c3d9309973b5c in term 1.
I20260810 03:51:24.666833 14393 leader_election.cc:302] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f1c3f1df7e4ad0b55c4c9f48dd88a5, ca4253aefefc40a98f9c3d9309973b5c; no voters:
I20260810 03:51:24.667011 15101 raft_consensus.cc:2781] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:51:24.667245 15101 raft_consensus.cc:686] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c [term 1 LEADER]: Becoming Leader. State: Replica: ca4253aefefc40a98f9c3d9309973b5c, State: Running, Role: LEADER
I20260810 03:51:24.667636 15101 consensus_queue.cc:237] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3d18e9ac344fd1ad737830758ac79d" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 38131 } } peers { permanent_uuid: "26f1c3f1df7e4ad0b55c4c9f48dd88a5" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 35725 } } peers { permanent_uuid: "ca4253aefefc40a98f9c3d9309973b5c" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 46193 } }
I20260810 03:51:24.670850 14248 catalog_manager.cc:5495] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c reported cstate change: term changed from 0 to 1, leader changed from <none> to ca4253aefefc40a98f9c3d9309973b5c (127.27.153.65). New cstate: current_term: 1 leader_uuid: "ca4253aefefc40a98f9c3d9309973b5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3d18e9ac344fd1ad737830758ac79d" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 38131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f1c3f1df7e4ad0b55c4c9f48dd88a5" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 35725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca4253aefefc40a98f9c3d9309973b5c" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 46193 } health_report { overall_health: HEALTHY } } }
I20260810 03:51:24.682827 15086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:24.679235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47486 (local address 127.27.153.65:46193)
0810 03:51:24.679365 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 03:51:24.679369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:51:24.679384 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:51:24.679536 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:51:24.679539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:51:24.679596 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:51:24.679696 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:51:24.679957 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.680462 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.681312 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:51:24.681526 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:51:24.681614 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:51:24.681963 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:51:24.682448 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:51:24.682548 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 03:51:24.682613 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 03:51:24.682679 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 03:51:24.688962 14973 raft_consensus.cc:1261] T e3044319fce04ee597334b74efd18a45 P 8f3d18e9ac344fd1ad737830758ac79d [term 1 FOLLOWER]: Refusing update from remote peer ca4253aefefc40a98f9c3d9309973b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:51:24.688962 14742 raft_consensus.cc:1261] T e3044319fce04ee597334b74efd18a45 P 26f1c3f1df7e4ad0b55c4c9f48dd88a5 [term 1 FOLLOWER]: Refusing update from remote peer ca4253aefefc40a98f9c3d9309973b5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:51:24.689770 15115 consensus_queue.cc:1035] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f3d18e9ac344fd1ad737830758ac79d" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 38131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:51:24.689978 15101 consensus_queue.cc:1035] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f1c3f1df7e4ad0b55c4c9f48dd88a5" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 35725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:51:24.699153 15122 mvcc.cc:205] Tried to move back new op lower bound from 7316823591677722624 to 7316823591601496064. Current Snapshot: MvccSnapshot[applied={T|T < 7316823591677722624}]
I20260810 03:51:24.710817 14248 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:51:24.711030 14248 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:51:24.713721 14248 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5fa1ed16d9f34fd3a2025f2223ee0e3e: Sending DeleteTablet for 3 replicas of tablet e3044319fce04ee597334b74efd18a45
I20260810 03:51:24.714634 14940 tablet_service.cc:1514] Processing DeleteTablet for tablet e3044319fce04ee597334b74efd18a45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:51:24 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:51456
I20260810 03:51:24.714679 14472 tablet_service.cc:1514] Processing DeleteTablet for tablet e3044319fce04ee597334b74efd18a45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:51:24 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:47480
I20260810 03:51:24.714715 14701 tablet_service.cc:1514] Processing DeleteTablet for tablet e3044319fce04ee597334b74efd18a45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:51:24 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:34480
I20260810 03:51:24.715327 15138 tablet_replica.cc:330] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c: stopping tablet replica
I20260810 03:51:24.715371 15140 tablet_replica.cc:330] T e3044319fce04ee597334b74efd18a45 P 26f1c3f1df7e4ad0b55c4c9f48dd88a5: stopping tablet replica
I20260810 03:51:24.715610 15139 tablet_replica.cc:330] T e3044319fce04ee597334b74efd18a45 P 8f3d18e9ac344fd1ad737830758ac79d: stopping tablet replica
I20260810 03:51:24.715934 15140 raft_consensus.cc:2229] T e3044319fce04ee597334b74efd18a45 P 26f1c3f1df7e4ad0b55c4c9f48dd88a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:51:24.715938 15139 raft_consensus.cc:2229] T e3044319fce04ee597334b74efd18a45 P 8f3d18e9ac344fd1ad737830758ac79d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:51:24.716280 15140 raft_consensus.cc:2258] T e3044319fce04ee597334b74efd18a45 P 26f1c3f1df7e4ad0b55c4c9f48dd88a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:51:24.716280 15139 raft_consensus.cc:2258] T e3044319fce04ee597334b74efd18a45 P 8f3d18e9ac344fd1ad737830758ac79d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:51:24.717109 15138 raft_consensus.cc:2229] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:51:24.717548 15138 raft_consensus.cc:2258] T e3044319fce04ee597334b74efd18a45 P ca4253aefefc40a98f9c3d9309973b5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:51:24.717932 15140 ts_tablet_manager.cc:1905] T e3044319fce04ee597334b74efd18a45 P 26f1c3f1df7e4ad0b55c4c9f48dd88a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:51:24.718077 15139 ts_tablet_manager.cc:1905] T e3044319fce04ee597334b74efd18a45 P 8f3d18e9ac344fd1ad737830758ac79d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:51:24.720654 15140 ts_tablet_manager.cc:1918] T e3044319fce04ee597334b74efd18a45 P 26f1c3f1df7e4ad0b55c4c9f48dd88a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:51:24.720763 15140 log.cc:1196] T e3044319fce04ee597334b74efd18a45 P 26f1c3f1df7e4ad0b55c4c9f48dd88a5: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/e3044319fce04ee597334b74efd18a45
I20260810 03:51:24.720956 15139 ts_tablet_manager.cc:1918] T e3044319fce04ee597334b74efd18a45 P 8f3d18e9ac344fd1ad737830758ac79d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:51:24.721062 15140 ts_tablet_manager.cc:1939] T e3044319fce04ee597334b74efd18a45 P 26f1c3f1df7e4ad0b55c4c9f48dd88a5: Deleting consensus metadata
I20260810 03:51:24.721168 15139 log.cc:1196] T e3044319fce04ee597334b74efd18a45 P 8f3d18e9ac344fd1ad737830758ac79d: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/e3044319fce04ee597334b74efd18a45
I20260810 03:51:24.721542 15139 ts_tablet_manager.cc:1939] T e3044319fce04ee597334b74efd18a45 P 8f3d18e9ac344fd1ad737830758ac79d: Deleting consensus metadata
I20260810 03:51:24.722347 14226 catalog_manager.cc:4841] TS 26f1c3f1df7e4ad0b55c4c9f48dd88a5 (127.27.153.66:35725): tablet e3044319fce04ee597334b74efd18a45 (table test-table [id=06e6f4f92d6244c8badb13f01d0bdf9d]) successfully deleted
I20260810 03:51:24.722833 14226 catalog_manager.cc:4841] TS 8f3d18e9ac344fd1ad737830758ac79d (127.27.153.67:38131): tablet e3044319fce04ee597334b74efd18a45 (table test-table [id=06e6f4f92d6244c8badb13f01d0bdf9d]) successfully deleted
I20260810 03:51:24.722935 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 14352
W20260810 03:51:24.745687 14224 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.153.65:46193 (error 108)
I20260810 03:51:24.745765 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 14590
I20260810 03:51:24.755191 15077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:51:24.754838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:46193 (local address 127.0.0.1:47494)
0810 03:51:24.755031 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 03:51:24.755099 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:46193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260810 03:51:24.755367 14224 catalog_manager.cc:4584] TS ca4253aefefc40a98f9c3d9309973b5c (127.27.153.65:46193): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e3044319fce04ee597334b74efd18a45: Network error: Client connection negotiation failed: client connection to 127.27.153.65:46193: connect: Connection refused (error 111)
I20260810 03:51:24.766850 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 14827
I20260810 03:51:24.779899 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 14189
2026-08-10T03:51:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2594 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:51:24 dist-test-slave-pww0 krb5kdc[15145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:51:24 dist-test-slave-pww0 krb5kdc[15145](info): set up 2 sockets
Aug 10 03:51:24 dist-test-slave-pww0 krb5kdc[15145](info): commencing operation
krb5kdc: starting...
W20260810 03:51:25.981612 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:51:26 dist-test-slave-pww0 krb5kdc[15145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:51:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:51:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.153.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.153.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.153.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/rangeradmin_127.27.153.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.153.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/rangeradmin_127.27.153.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.153.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.153.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.153.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/rangerlookup_127.27.153.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.153.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/rangerlookup_127.27.153.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.153.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.153.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.153.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/HTTP_127.27.153.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.153.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/HTTP_127.27.153.84@KRBTEST.COM.keytab.
I20260810 03:51:26.168778 28261 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskM5UkAJ/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/postgres -l logfile start
2026-08-10 03:51:28.711 UTC [15199] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 03:51:28.711 UTC [15199] LOG: listening on IPv4 address "127.27.153.84", port 49243
2026-08-10 03:51:28.715 UTC [15199] LOG: listening on Unix socket "/tmp/.s.PGSQL.49243"
2026-08-10 03:51:28.744 UTC [15202] LOG: database system was shut down at 2026-08-10 03:51:27 UTC
2026-08-10 03:51:28.748 UTC [15199] LOG: database system is ready to accept connections
I20260810 03:51:29.850939 28261 mini_postgres.cc:96] Postgres bound to 49243
2026-08-10 03:51:29.859 UTC [15223] FATAL: database "slave" does not exist
127.27.153.84:49243 - accepting connections
I20260810 03:51:29.860347 28261 mini_ranger.cc:161] Starting Ranger...
I20260810 03:51:29.877321 28261 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 03:51:30.375864 28261 mini_ranger.cc:87] Created ranger Postgres database
I20260810 03:51:30.375963 28261 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ranger-admin
2026-08-10 03:51:30,716 [I] DB FLAVOR :POSTGRES
2026-08-10 03:51:30,717 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 03:51:30,717 [I] Checking connection..
2026-08-10 03:51:30,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:49243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 03:51:30,976 [I] Checking connection passed.
2026-08-10 03:51:30,976 [I] --------- Verifying version history table ---------
2026-08-10 03:51:30,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:49243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:51:31,254 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 03:51:31,254 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:49243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:51:31,522 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 03:51:31,523 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 03:51:31,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:49243/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 03:51:31.775 UTC [15340] WARNING: there is no transaction in progress
2026-08-10 03:51:31,782 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 03:51:31,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:49243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:51:32,057 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 03:51:32,057 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 03:51:32,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:49243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 03:51:32,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:49243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 03:51:32,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:49243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pww0.c.gcp-upstream.internal','N') ;"
2026-08-10 03:51:32,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:49243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 03:51:33,063 [I] Table x_portal_user does not exist in database ranger
2026-08-10 03:51:33,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:49243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 03:51:33,321 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 03:51:33,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:49243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 03:51:33,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:49243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 03:51:33,840 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 03:51:33,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:49243/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 03:51:34.835 UTC [15524] WARNING: there is no transaction in progress
2026-08-10 03:51:34.852 UTC [15524] WARNING: there is no transaction in progress
2026-08-10 03:51:34.869 UTC [15524] WARNING: there is no transaction in progress
2026-08-10 03:51:34.885 UTC [15524] WARNING: there is no transaction in progress
2026-08-10 03:51:34.902 UTC [15524] WARNING: there is no transaction in progress
2026-08-10 03:51:34.924 UTC [15524] WARNING: there is no transaction in progress
2026-08-10 03:51:34.982 UTC [15524] WARNING: there is no transaction in progress
2026-08-10 03:51:34.990 UTC [15524] WARNING: there is no transaction in progress
2026-08-10 03:51:35.006 UTC [15524] WARNING: there is no transaction in progress
2026-08-10 03:51:35.017 UTC [15524] WARNING: there is no transaction in progress
2026-08-10 03:51:35.027 UTC [15524] WARNING: there is no transaction in progress
2026-08-10 03:51:35.041 UTC [15524] WARNING: there is no transaction in progress
2026-08-10 03:51:35.051 UTC [15524] WARNING: there is no transaction in progress
2026-08-10 03:51:35.058 UTC [15524] WARNING: there is no transaction in progress
2026-08-10 03:51:35.065 UTC [15524] WARNING: there is no transaction in progress
2026-08-10 03:51:35.073 UTC [15524] WARNING: there is no transaction in progress
2026-08-10 03:51:35,784 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 03:51:35,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:49243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 03:51:36,045 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 03:51:36,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:49243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 03:51:36,309 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 03:51:36,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:49243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 03:51:36,578 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 03:51:36,579 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:49243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 03:51:36,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:49243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 03:51:37,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:49243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pww0.c.gcp-upstream.internal';"
2026-08-10 03:51:37,325 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 03:51:37,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:49243/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 03:51:37,587 [I] DB_PATCHES have already been applied
I20260810 03:51:37.592907 28261 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ranger-admin:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskM5UkAJ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 03:51:37.592978 28261 mini_ranger.cc:193] Using host: 127.27.153.84
I20260810 03:51:37.597493 28261 mini_ranger.cc:227] Ranger admin URL: http://127.27.153.84:40399
Aug 10, 2026 3:51:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:51:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:51:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 3:51:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 3:51:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 3:51:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 3:51:38 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 03:51:38.863250 28261 mini_ranger.cc:160] Time spent starting Ranger: real 9.003s user 0.007s sys 0.006s
Aug 10, 2026 3:52:01 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 3:52:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 10, 2026 3:52:01 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 3:52:01 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 3:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 3:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 3:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 3:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 3:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 3:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 3:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 3:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 3:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 3:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 3:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 3:52:01 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 3:52:01 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 3:52:02 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 03:52:02.864048 28261 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.153.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.153.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.153.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/rangerkms_127.27.153.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.153.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/rangerkms_127.27.153.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.153.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.153.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.153.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/HTTP_127.27.153.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.153.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/HTTP_127.27.153.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 03:52:02 dist-test-slave-pww0 krb5kdc[15145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333922, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 03:52:02.955266 28261 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 03:52:02.967077 28261 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 03:52:03.816478 28261 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 03:52:03.816581 28261 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ranger-kms
I20260810 03:52:03.816692 28261 mini_ranger_kms.cc:227] Using postgres at 127.27.153.84:49243
2026-08-10 03:52:04,076 [I] DB FLAVOR :POSTGRES
2026-08-10 03:52:04,076 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 03:52:04,076 [I] Checking connection
2026-08-10 03:52:04,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:49243/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 03:52:04,341 [I] connection success
2026-08-10 03:52:04,341 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 03:52:04,342 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 03:52:04,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:49243/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 03:52:04,628 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 03:52:04,628 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 03:52:04,628 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 03:52:04,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:49243/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 03:52:04,911 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 03:52:05.434993 28261 mini_ranger_kms.cc:310] Created kms service
I20260810 03:52:05.629050 28261 mini_ranger_kms.cc:326] Created kudu user
I20260810 03:52:05.673848 28261 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 03:52:05 dist-test-slave-pww0 krb5kdc[15145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333922, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.153.84@KRBTEST.COM
I20260810 03:52:06.316612 28261 mini_ranger_kms.cc:384] Added ranger policy
I20260810 03:52:06.316891 28261 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ranger-kms:/tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 03:52:06.316926 28261 mini_ranger_kms.cc:240] Using host: 127.27.153.84
I20260810 03:52:06.325838 28261 mini_ranger_kms.cc:276] Ranger KMS PID: 15848
I20260810 03:52:06.326114 28261 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.153.84:42307
Aug 10, 2026 3:52:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:52:07 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:52:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 3:52:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 3:52:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 3:52:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 03:52:09 dist-test-slave-pww0 krb5kdc[15145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786333929, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.153.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:52:09.305949 28261 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.351s user 0.026s sys 0.225s
I20260810 03:52:09.306094 28261 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 03:52:09.306209 28261 mini_ranger_kms.cc:401] 127.27.153.84:42307/kms/v1/keys
Aug 10 03:52:09 dist-test-slave-pww0 krb5kdc[15145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333929, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.153.84@KRBTEST.COM for HTTP/127.27.153.84@KRBTEST.COM
Aug 10 03:52:09 dist-test-slave-pww0 krb5kdc[15145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333929, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.153.84@KRBTEST.COM for HTTP/127.27.153.84@KRBTEST.COM
Aug 10 03:52:12 dist-test-slave-pww0 krb5kdc[15145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:52:12.370616 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:34897
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:34897
--ranger_config_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.153.84:42307/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:52:12.519048 15943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:52:12.519346 15943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:52:12.519387 15943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:52:12.523528 15943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:52:12.523605 15943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:52:12.523623 15943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:52:12.523638 15943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:52:12.523653 15943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:52:12.529484 15943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.153.84:42307/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:34897
--ranger_config_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:34897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:52:12.530687 15943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:52:12.531829 15943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:52:12.541529 15951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:52:12.541472 15952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:52:12.541664 15943 server_base.cc:1034] running on GCE node
W20260810 03:52:12.541692 15954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:52:12.542644 15943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:52:12.544339 15943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:52:12.545562 15943 hybrid_clock.cc:648] HybridClock initialized: now 1786333932545536 us; error 44 us; skew 500 ppm
Aug 10 03:52:12 dist-test-slave-pww0 krb5kdc[15145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333932, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:52:12.552888 15943 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:52:12.553967 15943 webserver.cc:460] Webserver started at http://127.27.153.126:36841/ using document root <none> and password file <none>
I20260810 03:52:12.554539 15943 fs_manager.cc:359] Metadata directory not provided
I20260810 03:52:12.554600 15943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:52:12.554790 15943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:52:12 dist-test-slave-pww0 krb5kdc[15145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333932, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for HTTP/127.27.153.84@KRBTEST.COM
I20260810 03:52:12.614776 15943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "307cdcc454d4426f9c7730179a9d457f"
format_stamp: "Formatted at 2026-08-10 03:52:12 on dist-test-slave-pww0"
server_key: "37fc6fb0f40a80df125f4d289e697567"
server_key_iv: "7c3b924f6ccbcfc9f549aa3e1ae7a7cc"
server_key_version: "kuduclusterkey@0"
I20260810 03:52:12.615667 15943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "307cdcc454d4426f9c7730179a9d457f"
format_stamp: "Formatted at 2026-08-10 03:52:12 on dist-test-slave-pww0"
server_key: "37fc6fb0f40a80df125f4d289e697567"
server_key_iv: "7c3b924f6ccbcfc9f549aa3e1ae7a7cc"
server_key_version: "kuduclusterkey@0"
I20260810 03:52:12.621148 15943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 03:52:12.638022 15964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:52:12.639839 15943 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:52:12.640017 15943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "307cdcc454d4426f9c7730179a9d457f"
format_stamp: "Formatted at 2026-08-10 03:52:12 on dist-test-slave-pww0"
server_key: "37fc6fb0f40a80df125f4d289e697567"
server_key_iv: "7c3b924f6ccbcfc9f549aa3e1ae7a7cc"
server_key_version: "kuduclusterkey@0"
I20260810 03:52:12.640167 15943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:52:12.660550 15943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:52:12.664083 15943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:52:12.664310 15943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:52:12.673518 15943 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:34897
I20260810 03:52:12.673527 16074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:34897 every 8 connection(s)
I20260810 03:52:12.674675 15943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:52:12.678421 16075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:52:12.681466 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 15943
I20260810 03:52:12.681598 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:52:12.681896 28261 external_mini_cluster.cc:1428] Setting key 1dd6459ade20aaf538756702b4435f4d
I20260810 03:52:12.686414 16075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f: Bootstrap starting.
I20260810 03:52:12.689567 16075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f: Neither blocks nor log segments found. Creating new log.
I20260810 03:52:12.690585 16075 log.cc:824] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:52:12 dist-test-slave-pww0 krb5kdc[15145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:52:12.693102 16075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f: No bootstrap required, opened a new log
I20260810 03:52:12.697286 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:12.683537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46608 (local address 127.27.153.126:34897)
0810 03:52:12.684092 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0810 03:52:12.684106 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:52:12.684145 (+ 39us) server_negotiation.cc:407] Connection header received
0810 03:52:12.684955 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:52:12.684976 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:52:12.685435 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:52:12.685789 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:52:12.686677 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:52:12.687800 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:52:12.688691 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:52:12.689188 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:52:12.689366 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:52:12.691999 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:52:12.692005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:52:12.692020 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:52:12.692049 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:52:12.694174 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:52:12.694625 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:52:12.694630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:52:12.694635 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:52:12.694751 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:52:12.695033 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:52:12.695036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:52:12.695037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:52:12.695489 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:52:12.695687 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 03:52:12.696009 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 03:52:12.696468 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":143,"threads_started":1}
I20260810 03:52:12.697254 16075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307cdcc454d4426f9c7730179a9d457f" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 34897 } }
I20260810 03:52:12.697574 16075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:52:12.697636 16075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307cdcc454d4426f9c7730179a9d457f, State: Initialized, Role: FOLLOWER
I20260810 03:52:12.698149 16075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307cdcc454d4426f9c7730179a9d457f" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 34897 } }
I20260810 03:52:12.698324 16075 raft_consensus.cc:388] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:52:12.698410 16075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:52:12.698568 16075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:52:12.699774 16075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307cdcc454d4426f9c7730179a9d457f" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 34897 } }
I20260810 03:52:12.700148 16075 leader_election.cc:302] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 307cdcc454d4426f9c7730179a9d457f; no voters:
I20260810 03:52:12.700541 16075 leader_election.cc:288] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:52:12.700644 16086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:52:12.700907 16086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f [term 1 LEADER]: Becoming Leader. State: Replica: 307cdcc454d4426f9c7730179a9d457f, State: Running, Role: LEADER
I20260810 03:52:12.701292 16086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307cdcc454d4426f9c7730179a9d457f" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 34897 } }
I20260810 03:52:12.701864 16075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:52:12.702638 16086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "307cdcc454d4426f9c7730179a9d457f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307cdcc454d4426f9c7730179a9d457f" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 34897 } } }
I20260810 03:52:12.702858 16086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f [sys.catalog]: This master's current role is: LEADER
I20260810 03:52:12.703118 16086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 307cdcc454d4426f9c7730179a9d457f. Latest consensus state: current_term: 1 leader_uuid: "307cdcc454d4426f9c7730179a9d457f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307cdcc454d4426f9c7730179a9d457f" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 34897 } } }
I20260810 03:52:12.703260 16086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f [sys.catalog]: This master's current role is: LEADER
I20260810 03:52:12.704375 16075 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 03:52:12.708458 16105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:52:12.713205 16105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:52:12.723786 16105 catalog_manager.cc:1347] Generated new cluster ID: 0c79ef2be5c44f008b2734ea95d1f26d
I20260810 03:52:12.723966 16105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:52:12.738602 16105 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:52:12.740092 16105 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:52:12.750771 16105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f: Generated new TSK 0
I20260810 03:52:12.751865 16105 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.153.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
I20260810 03:52:13.479156 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.153.84:42307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:34897
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:52:13.629792 16165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:52:13.630244 16165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:52:13.630371 16165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:52:13.635694 16165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:52:13.635779 16165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:52:13.635866 16165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:52:13.642189 16165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.153.84:42307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:34897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:52:13.643761 16165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:52:13.645210 16165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:52:13.656859 16180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:52:13.657021 16174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:52:13.656872 16175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:52:13.657932 16165 server_base.cc:1034] running on GCE node
I20260810 03:52:13.658427 16165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:52:13.659106 16165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:52:13.660333 16165 hybrid_clock.cc:648] HybridClock initialized: now 1786333933660248 us; error 103 us; skew 500 ppm
Aug 10 03:52:13 dist-test-slave-pww0 krb5kdc[15145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333933, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:52:13.663621 16165 init.cc:377] Logged in from keytab as kudu/127.27.153.65@KRBTEST.COM (short username kudu)
I20260810 03:52:13.664765 16165 webserver.cc:460] Webserver started at http://127.27.153.65:45273/ using document root <none> and password file <none>
I20260810 03:52:13.665388 16165 fs_manager.cc:359] Metadata directory not provided
I20260810 03:52:13.665468 16165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:52:13.665722 16165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:52:13 dist-test-slave-pww0 krb5kdc[15145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333933, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for HTTP/127.27.153.84@KRBTEST.COM
I20260810 03:52:13.690265 16165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "291d41adc00445989379fd80ef909177"
format_stamp: "Formatted at 2026-08-10 03:52:13 on dist-test-slave-pww0"
server_key: "86653032279c69666a72c020e1d9b7d1"
server_key_iv: "fb3e3819ffa4131d27261600c7b0f8c6"
server_key_version: "kuduclusterkey@0"
I20260810 03:52:13.691350 16165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "291d41adc00445989379fd80ef909177"
format_stamp: "Formatted at 2026-08-10 03:52:13 on dist-test-slave-pww0"
server_key: "86653032279c69666a72c020e1d9b7d1"
server_key_iv: "fb3e3819ffa4131d27261600c7b0f8c6"
server_key_version: "kuduclusterkey@0"
I20260810 03:52:13.696985 16165 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 03:52:13.718816 16193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:52:13.720583 16165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:52:13.720749 16165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "291d41adc00445989379fd80ef909177"
format_stamp: "Formatted at 2026-08-10 03:52:13 on dist-test-slave-pww0"
server_key: "86653032279c69666a72c020e1d9b7d1"
server_key_iv: "fb3e3819ffa4131d27261600c7b0f8c6"
server_key_version: "kuduclusterkey@0"
I20260810 03:52:13.720914 16165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:52:13.736003 16165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:52:13.739912 16165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:52:13.740121 16165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:52:13.740885 16165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:52:13.742007 16165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:52:13.742075 16165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:52:13.742154 16165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:52:13.742187 16165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:52:13.757865 16165 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:43015
I20260810 03:52:13.759218 16165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:52:13.760524 16400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:43015 every 8 connection(s)
I20260810 03:52:13.768317 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 16165
I20260810 03:52:13.768455 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:52:13.768783 28261 external_mini_cluster.cc:1428] Setting key ac4f1a180db6434c4058ea0acbf39dfb
Aug 10 03:52:13 dist-test-slave-pww0 krb5kdc[15145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333933, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:52:13.784981 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:13.762963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:34897 (local address 127.27.153.65:47843)
0810 03:52:13.764671 (+ 1708us) negotiation.cc:107] Waiting for socket to connect
0810 03:52:13.764743 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 03:52:13.765862 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:52:13.766939 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:52:13.766949 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:52:13.767336 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:52:13.769392 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:52:13.769421 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:52:13.773046 (+ 3625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:52:13.773050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:52:13.774750 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:52:13.774767 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:52:13.775731 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:52:13.775736 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:52:13.775973 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:52:13.776764 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:52:13.776790 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:52:13.778712 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:52:13.781213 (+ 2501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:52:13.781221 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:52:13.781240 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:52:13.781633 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:52:13.782028 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:52:13.782031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:52:13.782034 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:52:13.782270 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:52:13.782834 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:52:13.782842 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:52:13.783226 (+ 384us) client_negotiation.cc:769] Sending connection context
0810 03:52:13.783496 (+ 270us) client_negotiation.cc:240] Negotiation successful
0810 03:52:13.783870 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1433,"thread_start_us":287,"threads_started":1}
I20260810 03:52:13.785866 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:13.762467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:47843 (local address 127.27.153.126:34897)
0810 03:52:13.766367 (+ 3900us) server_negotiation.cc:206] Beginning negotiation
0810 03:52:13.766372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:52:13.766389 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:52:13.766432 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:52:13.766436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:52:13.766487 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:52:13.766591 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:52:13.769576 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:52:13.772859 (+ 3283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:52:13.774943 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:52:13.775365 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:52:13.775511 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:52:13.778954 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:52:13.778959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:52:13.778962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:52:13.778995 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:52:13.781052 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:52:13.781781 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:52:13.781786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:52:13.781788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:52:13.781902 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:52:13.782403 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:52:13.782407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:52:13.782409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:52:13.782668 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:52:13.782771 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 03:52:13.785522 (+ 2751us) server_negotiation.cc:299] Negotiation successful
0810 03:52:13.785683 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3800,"thread_start_us":96,"threads_started":1}
I20260810 03:52:13.787937 16404 heartbeater.cc:347] Connected to a master server at 127.27.153.126:34897
I20260810 03:52:13.788224 16404 heartbeater.cc:464] Registering TS with master...
I20260810 03:52:13.788888 16404 heartbeater.cc:511] Master 127.27.153.126:34897 requested a full tablet report, sending...
I20260810 03:52:13.790967 16008 ts_manager.cc:194] Registered new tserver with Master: 291d41adc00445989379fd80ef909177 (127.27.153.65:43015)
I20260810 03:52:13.792646 16008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:47843
WARNING: no policy specified for kudu/127.27.153.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
I20260810 03:52:13.823622 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.153.84:42307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:34897
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:52:13 dist-test-slave-pww0 krb5kdc[15145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786333933, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 03:52:13.993510 16417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:52:13.993913 16417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:52:13.994505 16417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:52:14.000247 16417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:52:14.000326 16417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:52:14.000411 16417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:52:14.007824 16417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.153.84:42307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:34897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:52:14.009358 16417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:52:14.011034 16417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:52:14.028863 16431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:52:14.030023 16430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:52:14.031101 16434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:52:14.031616 16417 server_base.cc:1034] running on GCE node
I20260810 03:52:14.032292 16417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:52:14.033212 16417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:52:14.034525 16417 hybrid_clock.cc:648] HybridClock initialized: now 1786333934034451 us; error 97 us; skew 500 ppm
Aug 10 03:52:14 dist-test-slave-pww0 krb5kdc[15145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333934, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:52:14.038436 16417 init.cc:377] Logged in from keytab as kudu/127.27.153.66@KRBTEST.COM (short username kudu)
I20260810 03:52:14.039654 16417 webserver.cc:460] Webserver started at http://127.27.153.66:38995/ using document root <none> and password file <none>
I20260810 03:52:14.040410 16417 fs_manager.cc:359] Metadata directory not provided
I20260810 03:52:14.040512 16417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:52:14.040767 16417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:52:14 dist-test-slave-pww0 krb5kdc[15145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333934, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for HTTP/127.27.153.84@KRBTEST.COM
I20260810 03:52:14.076198 16417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "9d78169e736f4316a4cb71f2b2f26df7"
format_stamp: "Formatted at 2026-08-10 03:52:14 on dist-test-slave-pww0"
server_key: "7158ee27ccff448ded32c6a8a60e3dd3"
server_key_iv: "1348bf424c5894b920a017ac237717cf"
server_key_version: "kuduclusterkey@0"
I20260810 03:52:14.077153 16417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "9d78169e736f4316a4cb71f2b2f26df7"
format_stamp: "Formatted at 2026-08-10 03:52:14 on dist-test-slave-pww0"
server_key: "7158ee27ccff448ded32c6a8a60e3dd3"
server_key_iv: "1348bf424c5894b920a017ac237717cf"
server_key_version: "kuduclusterkey@0"
I20260810 03:52:14.082799 16417 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 03:52:14.106391 16444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:52:14.108206 16417 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260810 03:52:14.108359 16417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "9d78169e736f4316a4cb71f2b2f26df7"
format_stamp: "Formatted at 2026-08-10 03:52:14 on dist-test-slave-pww0"
server_key: "7158ee27ccff448ded32c6a8a60e3dd3"
server_key_iv: "1348bf424c5894b920a017ac237717cf"
server_key_version: "kuduclusterkey@0"
I20260810 03:52:14.108513 16417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:52:14.141629 16417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:52:14.145532 16417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:52:14.145740 16417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:52:14.146414 16417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:52:14.147746 16417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:52:14.147816 16417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:52:14.147878 16417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:52:14.147905 16417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:52:14.161926 16417 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:34059
I20260810 03:52:14.163272 16417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:52:14.165155 16652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:34059 every 8 connection(s)
I20260810 03:52:14.168149 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 16417
I20260810 03:52:14.168288 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:52:14.168612 28261 external_mini_cluster.cc:1428] Setting key 5b72c40de6d56ea7c718ec828c2417f9
Aug 10 03:52:14 dist-test-slave-pww0 krb5kdc[15145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333934, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:52:14.184583 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:14.169323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:45037 (local address 127.27.153.126:34897)
0810 03:52:14.169462 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 03:52:14.169467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:52:14.169988 (+ 521us) server_negotiation.cc:407] Connection header received
0810 03:52:14.171351 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:52:14.171355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:52:14.171410 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:52:14.171499 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:52:14.173323 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.174126 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.175930 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.176295 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.176381 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:52:14.179788 (+ 3407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:52:14.179792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:52:14.179795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:52:14.179829 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:52:14.181639 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:52:14.182384 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:52:14.182388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:52:14.182390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:52:14.182494 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:52:14.183013 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:52:14.183017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:52:14.183018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:52:14.183304 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:52:14.183391 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 03:52:14.184225 (+ 834us) server_negotiation.cc:299] Negotiation successful
0810 03:52:14.184386 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:52:14.186733 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:14.168002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:34897 (local address 127.27.153.66:45037)
0810 03:52:14.169802 (+ 1800us) negotiation.cc:107] Waiting for socket to connect
0810 03:52:14.169874 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 03:52:14.171061 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:52:14.171976 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:52:14.171986 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:52:14.172384 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:52:14.173113 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:52:14.173136 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.174322 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.174326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:52:14.175766 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:52:14.175779 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.176617 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.176620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:52:14.176858 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:52:14.177647 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:52:14.177671 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:52:14.179535 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:52:14.181802 (+ 2267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:52:14.181809 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:52:14.181829 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:52:14.182246 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:52:14.182608 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:52:14.182610 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:52:14.182612 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:52:14.182896 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:52:14.183502 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:52:14.183511 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:52:14.183911 (+ 400us) client_negotiation.cc:769] Sending connection context
0810 03:52:14.185184 (+ 1273us) client_negotiation.cc:240] Negotiation successful
0810 03:52:14.185601 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1540,"thread_start_us":185,"threads_started":1}
I20260810 03:52:14.190145 16653 heartbeater.cc:347] Connected to a master server at 127.27.153.126:34897
I20260810 03:52:14.190493 16653 heartbeater.cc:464] Registering TS with master...
I20260810 03:52:14.191231 16653 heartbeater.cc:511] Master 127.27.153.126:34897 requested a full tablet report, sending...
I20260810 03:52:14.192682 16008 ts_manager.cc:194] Registered new tserver with Master: 9d78169e736f4316a4cb71f2b2f26df7 (127.27.153.66:34059)
I20260810 03:52:14.193527 16008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:45037
WARNING: no policy specified for kudu/127.27.153.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
I20260810 03:52:14.227211 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.153.84:42307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:34897
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:43355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:52:14.396483 16669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:52:14.396795 16669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:52:14.396859 16669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:52:14.401597 16669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:52:14.401677 16669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:52:14.401764 16669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:52:14.407603 16669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:43355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.153.84:42307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:34897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:52:14.408818 16669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:52:14.409922 16669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:52:14.421669 16681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:52:14.422204 16678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:52:14.423432 16669 server_base.cc:1034] running on GCE node
W20260810 03:52:14.426549 16676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:52:14.427099 16669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:52:14.428000 16669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:52:14.429181 16669 hybrid_clock.cc:648] HybridClock initialized: now 1786333934429108 us; error 93 us; skew 500 ppm
Aug 10 03:52:14 dist-test-slave-pww0 krb5kdc[15145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333934, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:52:14.434942 16669 init.cc:377] Logged in from keytab as kudu/127.27.153.67@KRBTEST.COM (short username kudu)
I20260810 03:52:14.436239 16669 webserver.cc:460] Webserver started at http://127.27.153.67:35213/ using document root <none> and password file <none>
I20260810 03:52:14.436976 16669 fs_manager.cc:359] Metadata directory not provided
I20260810 03:52:14.437057 16669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:52:14.437302 16669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:52:14 dist-test-slave-pww0 krb5kdc[15145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333934, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for HTTP/127.27.153.84@KRBTEST.COM
I20260810 03:52:14.454782 16669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "98a75889a2774502b1420456689c3856"
format_stamp: "Formatted at 2026-08-10 03:52:14 on dist-test-slave-pww0"
server_key: "dc08a7ca8bfc73480fe3c31b8aaeed01"
server_key_iv: "524e7b1ad651ed5d7ac43d07ef39d746"
server_key_version: "kuduclusterkey@0"
I20260810 03:52:14.455699 16669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "98a75889a2774502b1420456689c3856"
format_stamp: "Formatted at 2026-08-10 03:52:14 on dist-test-slave-pww0"
server_key: "dc08a7ca8bfc73480fe3c31b8aaeed01"
server_key_iv: "524e7b1ad651ed5d7ac43d07ef39d746"
server_key_version: "kuduclusterkey@0"
Aug 10 03:52:14 dist-test-slave-pww0 krb5kdc[15145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333933, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for HTTP/127.27.153.84@KRBTEST.COM
I20260810 03:52:14.477324 16669 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.005s sys 0.000s
I20260810 03:52:14.500587 16691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:52:14.502350 16669 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260810 03:52:14.502521 16669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "98a75889a2774502b1420456689c3856"
format_stamp: "Formatted at 2026-08-10 03:52:14 on dist-test-slave-pww0"
server_key: "dc08a7ca8bfc73480fe3c31b8aaeed01"
server_key_iv: "524e7b1ad651ed5d7ac43d07ef39d746"
server_key_version: "kuduclusterkey@0"
I20260810 03:52:14.502686 16669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:52:14.527026 16669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:52:14.530670 16669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:52:14.530993 16669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:52:14.531888 16669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:52:14.533037 16669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:52:14.533151 16669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:52:14.533448 16669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:52:14.533520 16669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:52:14.544951 16669 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:46857
I20260810 03:52:14.544988 16911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:46857 every 8 connection(s)
I20260810 03:52:14.546126 16669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:52:14.551364 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 16669
I20260810 03:52:14.551534 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:52:14.551857 28261 external_mini_cluster.cc:1428] Setting key f6228de0a1d6596225c9e931a084c72b
Aug 10 03:52:14 dist-test-slave-pww0 krb5kdc[15145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333934, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:52:14.560688 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:14.548168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:59505 (local address 127.27.153.126:34897)
0810 03:52:14.548353 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 03:52:14.548359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:52:14.549298 (+ 939us) server_negotiation.cc:407] Connection header received
0810 03:52:14.550331 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:52:14.550335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:52:14.550395 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:52:14.550469 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:52:14.551899 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.552600 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.554301 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.554525 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.554612 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:52:14.557123 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:52:14.557126 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:52:14.557129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:52:14.557155 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:52:14.558493 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:52:14.559013 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:52:14.559016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:52:14.559017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:52:14.559094 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:52:14.559428 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:52:14.559432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:52:14.559433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:52:14.559705 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:52:14.559795 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 03:52:14.560376 (+ 581us) server_negotiation.cc:299] Negotiation successful
0810 03:52:14.560510 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 03:52:14.561733 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:14.548610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:34897 (local address 127.27.153.67:59505)
0810 03:52:14.549129 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0810 03:52:14.549186 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 03:52:14.550094 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:52:14.550660 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:52:14.550674 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:52:14.550953 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:52:14.551711 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:52:14.551737 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.552838 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.552841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:52:14.554184 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:52:14.554194 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.554699 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.554704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:52:14.554869 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:52:14.555440 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:52:14.555486 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:52:14.556949 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:52:14.558611 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:52:14.558618 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:52:14.558637 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:52:14.558914 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:52:14.559179 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:52:14.559182 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:52:14.559185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:52:14.559339 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:52:14.559851 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:52:14.559857 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:52:14.560123 (+ 266us) client_negotiation.cc:769] Sending connection context
0810 03:52:14.560314 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 03:52:14.560661 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":144,"threads_started":1}
I20260810 03:52:14.562873 16912 heartbeater.cc:347] Connected to a master server at 127.27.153.126:34897
I20260810 03:52:14.563129 16912 heartbeater.cc:464] Registering TS with master...
I20260810 03:52:14.563709 16912 heartbeater.cc:511] Master 127.27.153.126:34897 requested a full tablet report, sending...
I20260810 03:52:14.565029 16003 ts_manager.cc:194] Registered new tserver with Master: 98a75889a2774502b1420456689c3856 (127.27.153.67:46857)
I20260810 03:52:14.565778 16003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:59505
I20260810 03:52:14.565871 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:52:14.576532 16411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:14.569399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46612 (local address 127.27.153.126:34897)
0810 03:52:14.569535 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 03:52:14.569540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:52:14.569683 (+ 143us) server_negotiation.cc:407] Connection header received
0810 03:52:14.569823 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:52:14.569825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:52:14.569866 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:52:14.569926 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:52:14.570641 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.571139 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.572015 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.572237 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.572323 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:52:14.573616 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:52:14.573619 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:52:14.573621 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:52:14.573645 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:52:14.574933 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:52:14.575341 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:52:14.575345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:52:14.575346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:52:14.575422 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:52:14.575729 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:52:14.575732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:52:14.575734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:52:14.575911 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:52:14.576030 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 03:52:14.576263 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 03:52:14.576376 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:52:14.579706 16003 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:52:14.582826 16003 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:52:14.603871 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:14.594555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:46857 (local address 127.0.0.1:52738)
0810 03:52:14.596879 (+ 2324us) negotiation.cc:107] Waiting for socket to connect
0810 03:52:14.596890 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 03:52:14.596957 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:52:14.598219 (+ 1262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:52:14.598222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:52:14.598241 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:52:14.598365 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:52:14.598371 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.599994 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.599996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:52:14.601273 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:52:14.601293 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.603149 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.603153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:52:14.603309 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:52:14.603361 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 03:52:14.603539 (+ 178us) client_negotiation.cc:240] Negotiation successful
0810 03:52:14.603696 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2274,"mutex_wait_us":1,"thread_start_us":117,"threads_started":1}
I20260810 03:52:14.604543 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:14.594774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52738 (local address 127.27.153.67:46857)
0810 03:52:14.595903 (+ 1129us) server_negotiation.cc:206] Beginning negotiation
0810 03:52:14.595914 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:52:14.597051 (+ 1137us) server_negotiation.cc:407] Connection header received
0810 03:52:14.597138 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:52:14.597143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:52:14.597359 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:52:14.597504 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:52:14.598503 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.599725 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.601481 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.602774 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.602896 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:52:14.603083 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 03:52:14.604251 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0810 03:52:14.604395 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1029,"thread_start_us":142,"threads_started":1}
I20260810 03:52:14.606565 16935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:14.593689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:43015 (local address 127.0.0.1:38704)
0810 03:52:14.596184 (+ 2495us) negotiation.cc:107] Waiting for socket to connect
0810 03:52:14.596250 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 03:52:14.596415 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:52:14.597168 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:52:14.597176 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:52:14.597201 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:52:14.597348 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:52:14.597355 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.599772 (+ 2417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.599776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:52:14.604777 (+ 5001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:52:14.604789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.606126 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.606130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:52:14.606239 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:52:14.606263 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:52:14.606342 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 03:52:14.606422 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2302,"mutex_wait_us":3592,"thread_start_us":154,"threads_started":1}
I20260810 03:52:14.607095 16940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:14.595197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38704 (local address 127.27.153.65:43015)
0810 03:52:14.595700 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0810 03:52:14.595706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:52:14.596382 (+ 676us) server_negotiation.cc:407] Connection header received
0810 03:52:14.596572 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:52:14.596578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:52:14.596827 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:52:14.596982 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:52:14.597488 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.599618 (+ 2130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.604934 (+ 5316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.605813 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.605915 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:52:14.606060 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 03:52:14.606845 (+ 785us) server_negotiation.cc:299] Negotiation successful
0810 03:52:14.606957 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":95,"threads_started":1}
I20260810 03:52:14.607611 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:14.593738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:34059 (local address 127.0.0.1:56872)
0810 03:52:14.596644 (+ 2906us) negotiation.cc:107] Waiting for socket to connect
0810 03:52:14.596656 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:52:14.596732 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:52:14.599863 (+ 3131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:52:14.599866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:52:14.599889 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:52:14.602234 (+ 2345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:52:14.602246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.603822 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.603825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:52:14.604988 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:52:14.605000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.607200 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.607204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:52:14.607293 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:52:14.607316 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 03:52:14.607384 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 03:52:14.607452 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2848,"mutex_wait_us":1233,"thread_start_us":110,"threads_started":1}
I20260810 03:52:14.608160 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:14.594810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.27.153.66:34059)
0810 03:52:14.595247 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0810 03:52:14.595260 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:52:14.597651 (+ 2391us) server_negotiation.cc:407] Connection header received
0810 03:52:14.597735 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:52:14.597740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:52:14.597929 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:52:14.598071 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:52:14.602406 (+ 4335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.603679 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.605161 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.606885 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.606992 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:52:14.607134 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 03:52:14.607847 (+ 713us) server_negotiation.cc:299] Negotiation successful
0810 03:52:14.607966 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":110,"threads_started":1}
I20260810 03:52:14.608522 16789 tablet_service.cc:1467] Processing CreateTablet for tablet d73786d242ef42e488ea5f5d72731fc1 (DEFAULT_TABLE table=test-table [id=c30e4b535bbc4719aec077dc35066ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:52:14.609669 16789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73786d242ef42e488ea5f5d72731fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:52:14.610198 16260 tablet_service.cc:1467] Processing CreateTablet for tablet d73786d242ef42e488ea5f5d72731fc1 (DEFAULT_TABLE table=test-table [id=c30e4b535bbc4719aec077dc35066ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:52:14.611433 16260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73786d242ef42e488ea5f5d72731fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:52:14.617985 16531 tablet_service.cc:1467] Processing CreateTablet for tablet d73786d242ef42e488ea5f5d72731fc1 (DEFAULT_TABLE table=test-table [id=c30e4b535bbc4719aec077dc35066ada]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:52:14.619247 16531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d73786d242ef42e488ea5f5d72731fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:52:14.628903 16941 tablet_bootstrap.cc:492] T d73786d242ef42e488ea5f5d72731fc1 P 98a75889a2774502b1420456689c3856: Bootstrap starting.
I20260810 03:52:14.632593 16941 tablet_bootstrap.cc:654] T d73786d242ef42e488ea5f5d72731fc1 P 98a75889a2774502b1420456689c3856: Neither blocks nor log segments found. Creating new log.
I20260810 03:52:14.633730 16941 log.cc:824] T d73786d242ef42e488ea5f5d72731fc1 P 98a75889a2774502b1420456689c3856: Log is configured to *not* fsync() on all Append() calls
I20260810 03:52:14.635015 16943 tablet_bootstrap.cc:492] T d73786d242ef42e488ea5f5d72731fc1 P 291d41adc00445989379fd80ef909177: Bootstrap starting.
I20260810 03:52:14.637835 16943 tablet_bootstrap.cc:654] T d73786d242ef42e488ea5f5d72731fc1 P 291d41adc00445989379fd80ef909177: Neither blocks nor log segments found. Creating new log.
I20260810 03:52:14.638553 16947 tablet_bootstrap.cc:492] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7: Bootstrap starting.
I20260810 03:52:14.640254 16941 tablet_bootstrap.cc:492] T d73786d242ef42e488ea5f5d72731fc1 P 98a75889a2774502b1420456689c3856: No bootstrap required, opened a new log
I20260810 03:52:14.640479 16941 ts_tablet_manager.cc:1397] T d73786d242ef42e488ea5f5d72731fc1 P 98a75889a2774502b1420456689c3856: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260810 03:52:14.644217 16941 raft_consensus.cc:348] T d73786d242ef42e488ea5f5d72731fc1 P 98a75889a2774502b1420456689c3856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d78169e736f4316a4cb71f2b2f26df7" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34059 } } peers { permanent_uuid: "291d41adc00445989379fd80ef909177" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43015 } } peers { permanent_uuid: "98a75889a2774502b1420456689c3856" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46857 } }
I20260810 03:52:14.644479 16941 raft_consensus.cc:374] T d73786d242ef42e488ea5f5d72731fc1 P 98a75889a2774502b1420456689c3856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:52:14.644532 16941 raft_consensus.cc:729] T d73786d242ef42e488ea5f5d72731fc1 P 98a75889a2774502b1420456689c3856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98a75889a2774502b1420456689c3856, State: Initialized, Role: FOLLOWER
I20260810 03:52:14.645042 16941 consensus_queue.cc:260] T d73786d242ef42e488ea5f5d72731fc1 P 98a75889a2774502b1420456689c3856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d78169e736f4316a4cb71f2b2f26df7" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34059 } } peers { permanent_uuid: "291d41adc00445989379fd80ef909177" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43015 } } peers { permanent_uuid: "98a75889a2774502b1420456689c3856" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46857 } }
I20260810 03:52:14.645834 16912 heartbeater.cc:503] Master 127.27.153.126:34897 was elected leader, sending a full tablet report...
I20260810 03:52:14.646283 16941 ts_tablet_manager.cc:1428] T d73786d242ef42e488ea5f5d72731fc1 P 98a75889a2774502b1420456689c3856: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 03:52:14.657531 16943 log.cc:824] T d73786d242ef42e488ea5f5d72731fc1 P 291d41adc00445989379fd80ef909177: Log is configured to *not* fsync() on all Append() calls
I20260810 03:52:14.660725 16947 tablet_bootstrap.cc:654] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7: Neither blocks nor log segments found. Creating new log.
I20260810 03:52:14.662746 16943 tablet_bootstrap.cc:492] T d73786d242ef42e488ea5f5d72731fc1 P 291d41adc00445989379fd80ef909177: No bootstrap required, opened a new log
I20260810 03:52:14.664037 16947 log.cc:824] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7: Log is configured to *not* fsync() on all Append() calls
I20260810 03:52:14.664497 16943 ts_tablet_manager.cc:1397] T d73786d242ef42e488ea5f5d72731fc1 P 291d41adc00445989379fd80ef909177: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.000s
I20260810 03:52:14.670662 16943 raft_consensus.cc:348] T d73786d242ef42e488ea5f5d72731fc1 P 291d41adc00445989379fd80ef909177 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d78169e736f4316a4cb71f2b2f26df7" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34059 } } peers { permanent_uuid: "291d41adc00445989379fd80ef909177" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43015 } } peers { permanent_uuid: "98a75889a2774502b1420456689c3856" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46857 } }
I20260810 03:52:14.672453 16943 raft_consensus.cc:374] T d73786d242ef42e488ea5f5d72731fc1 P 291d41adc00445989379fd80ef909177 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:52:14.672600 16943 raft_consensus.cc:729] T d73786d242ef42e488ea5f5d72731fc1 P 291d41adc00445989379fd80ef909177 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 291d41adc00445989379fd80ef909177, State: Initialized, Role: FOLLOWER
I20260810 03:52:14.673243 16943 consensus_queue.cc:260] T d73786d242ef42e488ea5f5d72731fc1 P 291d41adc00445989379fd80ef909177 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d78169e736f4316a4cb71f2b2f26df7" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34059 } } peers { permanent_uuid: "291d41adc00445989379fd80ef909177" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43015 } } peers { permanent_uuid: "98a75889a2774502b1420456689c3856" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46857 } }
I20260810 03:52:14.674692 16943 ts_tablet_manager.cc:1428] T d73786d242ef42e488ea5f5d72731fc1 P 291d41adc00445989379fd80ef909177: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260810 03:52:14.675163 16404 heartbeater.cc:503] Master 127.27.153.126:34897 was elected leader, sending a full tablet report...
I20260810 03:52:14.678618 16947 tablet_bootstrap.cc:492] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7: No bootstrap required, opened a new log
I20260810 03:52:14.678957 16947 ts_tablet_manager.cc:1397] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.004s
I20260810 03:52:14.683728 16947 raft_consensus.cc:348] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d78169e736f4316a4cb71f2b2f26df7" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34059 } } peers { permanent_uuid: "291d41adc00445989379fd80ef909177" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43015 } } peers { permanent_uuid: "98a75889a2774502b1420456689c3856" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46857 } }
I20260810 03:52:14.684067 16947 raft_consensus.cc:374] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:52:14.684125 16947 raft_consensus.cc:729] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d78169e736f4316a4cb71f2b2f26df7, State: Initialized, Role: FOLLOWER
I20260810 03:52:14.684629 16947 consensus_queue.cc:260] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d78169e736f4316a4cb71f2b2f26df7" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34059 } } peers { permanent_uuid: "291d41adc00445989379fd80ef909177" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43015 } } peers { permanent_uuid: "98a75889a2774502b1420456689c3856" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46857 } }
I20260810 03:52:14.685619 16947 ts_tablet_manager.cc:1428] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 03:52:14.686007 16653 heartbeater.cc:503] Master 127.27.153.126:34897 was elected leader, sending a full tablet report...
W20260810 03:52:14.761359 16407 tablet.cc:2369] T d73786d242ef42e488ea5f5d72731fc1 P 291d41adc00445989379fd80ef909177: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:52:14.786155 16958 raft_consensus.cc:482] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:52:14.786417 16958 raft_consensus.cc:504] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d78169e736f4316a4cb71f2b2f26df7" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34059 } } peers { permanent_uuid: "291d41adc00445989379fd80ef909177" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43015 } } peers { permanent_uuid: "98a75889a2774502b1420456689c3856" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46857 } }
I20260810 03:52:14.789546 16958 leader_election.cc:288] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 291d41adc00445989379fd80ef909177 (127.27.153.65:43015), 98a75889a2774502b1420456689c3856 (127.27.153.67:46857)
I20260810 03:52:14.794695 16939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:14.789401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:43107 (local address 127.27.153.67:46857)
0810 03:52:14.789512 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 03:52:14.789517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:52:14.789908 (+ 391us) server_negotiation.cc:407] Connection header received
0810 03:52:14.790101 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:52:14.790104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:52:14.790153 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:52:14.790205 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:52:14.790603 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.791540 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.793392 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.793969 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.794064 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:52:14.794120 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 03:52:14.794465 (+ 345us) server_negotiation.cc:299] Negotiation successful
0810 03:52:14.794556 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 03:52:14.795186 16963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:14.788986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:46857 (local address 127.27.153.66:43107)
0810 03:52:14.789821 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0810 03:52:14.789834 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:52:14.789980 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:52:14.790350 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:52:14.790353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:52:14.790369 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:52:14.790482 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:52:14.790487 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.791748 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.791751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:52:14.793211 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:52:14.793225 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.794229 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.794232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:52:14.794332 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:52:14.794355 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 03:52:14.794932 (+ 577us) client_negotiation.cc:240] Negotiation successful
0810 03:52:14.795023 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":114,"threads_started":1}
I20260810 03:52:14.795890 16830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73786d242ef42e488ea5f5d72731fc1" candidate_uuid: "9d78169e736f4316a4cb71f2b2f26df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98a75889a2774502b1420456689c3856" is_pre_election: true
I20260810 03:52:14.796254 16830 raft_consensus.cc:2454] T d73786d242ef42e488ea5f5d72731fc1 P 98a75889a2774502b1420456689c3856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d78169e736f4316a4cb71f2b2f26df7 in term 0.
I20260810 03:52:14.796854 16451 leader_election.cc:302] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98a75889a2774502b1420456689c3856, 9d78169e736f4316a4cb71f2b2f26df7; no voters:
I20260810 03:52:14.797219 16958 raft_consensus.cc:2781] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:52:14.797314 16958 raft_consensus.cc:482] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:52:14.797366 16958 raft_consensus.cc:3037] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:52:14.798237 16940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:14.788502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:35661 (local address 127.27.153.65:43015)
0810 03:52:14.788630 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 03:52:14.788634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:52:14.788982 (+ 348us) server_negotiation.cc:407] Connection header received
0810 03:52:14.789254 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:52:14.789257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:52:14.789310 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:52:14.789358 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:52:14.789933 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.790722 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.796886 (+ 6164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.797447 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.797588 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:52:14.797646 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 03:52:14.797995 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 03:52:14.798095 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 03:52:14.798722 16959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:14.788045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:43015 (local address 127.27.153.66:35661)
0810 03:52:14.788861 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0810 03:52:14.788881 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 03:52:14.789104 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:52:14.789508 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:52:14.789511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:52:14.789531 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:52:14.789780 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:52:14.789791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.795530 (+ 5739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.795533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:52:14.796634 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:52:14.796648 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.797732 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.797737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:52:14.797826 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:52:14.797853 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 03:52:14.798486 (+ 633us) client_negotiation.cc:240] Negotiation successful
0810 03:52:14.798574 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":115,"threads_started":1}
I20260810 03:52:14.798794 16958 raft_consensus.cc:504] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d78169e736f4316a4cb71f2b2f26df7" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34059 } } peers { permanent_uuid: "291d41adc00445989379fd80ef909177" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43015 } } peers { permanent_uuid: "98a75889a2774502b1420456689c3856" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46857 } }
I20260810 03:52:14.799220 16958 leader_election.cc:288] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7 [CANDIDATE]: Term 1 election: Requested vote from peers 291d41adc00445989379fd80ef909177 (127.27.153.65:43015), 98a75889a2774502b1420456689c3856 (127.27.153.67:46857)
I20260810 03:52:14.799774 16285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73786d242ef42e488ea5f5d72731fc1" candidate_uuid: "9d78169e736f4316a4cb71f2b2f26df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "291d41adc00445989379fd80ef909177" is_pre_election: true
I20260810 03:52:14.800061 16285 raft_consensus.cc:2454] T d73786d242ef42e488ea5f5d72731fc1 P 291d41adc00445989379fd80ef909177 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d78169e736f4316a4cb71f2b2f26df7 in term 0.
I20260810 03:52:14.800417 16314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73786d242ef42e488ea5f5d72731fc1" candidate_uuid: "9d78169e736f4316a4cb71f2b2f26df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "291d41adc00445989379fd80ef909177"
I20260810 03:52:14.800519 16314 raft_consensus.cc:3037] T d73786d242ef42e488ea5f5d72731fc1 P 291d41adc00445989379fd80ef909177 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:52:14.801800 16314 raft_consensus.cc:2454] T d73786d242ef42e488ea5f5d72731fc1 P 291d41adc00445989379fd80ef909177 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d78169e736f4316a4cb71f2b2f26df7 in term 1.
W20260810 03:52:14.802268 16913 tablet.cc:2369] T d73786d242ef42e488ea5f5d72731fc1 P 98a75889a2774502b1420456689c3856: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:52:14.802317 16830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d73786d242ef42e488ea5f5d72731fc1" candidate_uuid: "9d78169e736f4316a4cb71f2b2f26df7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98a75889a2774502b1420456689c3856"
I20260810 03:52:14.802475 16830 raft_consensus.cc:3037] T d73786d242ef42e488ea5f5d72731fc1 P 98a75889a2774502b1420456689c3856 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:52:14.803059 16451 leader_election.cc:302] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 291d41adc00445989379fd80ef909177, 9d78169e736f4316a4cb71f2b2f26df7; no voters:
I20260810 03:52:14.803257 16958 raft_consensus.cc:2781] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:52:14.803486 16958 raft_consensus.cc:686] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7 [term 1 LEADER]: Becoming Leader. State: Replica: 9d78169e736f4316a4cb71f2b2f26df7, State: Running, Role: LEADER
I20260810 03:52:14.803857 16958 consensus_queue.cc:237] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d78169e736f4316a4cb71f2b2f26df7" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34059 } } peers { permanent_uuid: "291d41adc00445989379fd80ef909177" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43015 } } peers { permanent_uuid: "98a75889a2774502b1420456689c3856" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46857 } }
I20260810 03:52:14.804172 16830 raft_consensus.cc:2454] T d73786d242ef42e488ea5f5d72731fc1 P 98a75889a2774502b1420456689c3856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d78169e736f4316a4cb71f2b2f26df7 in term 1.
I20260810 03:52:14.808023 16003 catalog_manager.cc:5495] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d78169e736f4316a4cb71f2b2f26df7 (127.27.153.66). New cstate: current_term: 1 leader_uuid: "9d78169e736f4316a4cb71f2b2f26df7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d78169e736f4316a4cb71f2b2f26df7" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "291d41adc00445989379fd80ef909177" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98a75889a2774502b1420456689c3856" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46857 } health_report { overall_health: UNKNOWN } } }
Aug 10 03:52:14 dist-test-slave-pww0 krb5kdc[15145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333933, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for HTTP/127.27.153.84@KRBTEST.COM
I20260810 03:52:14.846966 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:14.839729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56874 (local address 127.27.153.66:34059)
0810 03:52:14.839875 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 03:52:14.839878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:52:14.839894 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:52:14.840132 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:52:14.840136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:52:14.840209 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:52:14.840339 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:52:14.841065 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.841906 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.844448 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:52:14.844827 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:52:14.844943 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:52:14.845475 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:52:14.846063 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:52:14.846198 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 03:52:14.846735 (+ 537us) server_negotiation.cc:299] Negotiation successful
0810 03:52:14.846811 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 03:52:14.859192 16830 raft_consensus.cc:1261] T d73786d242ef42e488ea5f5d72731fc1 P 98a75889a2774502b1420456689c3856 [term 1 FOLLOWER]: Refusing update from remote peer 9d78169e736f4316a4cb71f2b2f26df7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:52:14.859292 16314 raft_consensus.cc:1261] T d73786d242ef42e488ea5f5d72731fc1 P 291d41adc00445989379fd80ef909177 [term 1 FOLLOWER]: Refusing update from remote peer 9d78169e736f4316a4cb71f2b2f26df7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:52:14.860200 16958 consensus_queue.cc:1035] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98a75889a2774502b1420456689c3856" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 46857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:52:14.860796 16966 consensus_queue.cc:1035] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "291d41adc00445989379fd80ef909177" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:52:14.880084 16973 mvcc.cc:205] Tried to move back new op lower bound from 7316823797164765184 to 7316823796961456128. Current Snapshot: MvccSnapshot[applied={T|T < 7316823797164765184}]
I20260810 03:52:14.880404 16974 mvcc.cc:205] Tried to move back new op lower bound from 7316823797164765184 to 7316823796961456128. Current Snapshot: MvccSnapshot[applied={T|T < 7316823797164765184}]
I20260810 03:52:14.882997 16972 mvcc.cc:205] Tried to move back new op lower bound from 7316823797164765184 to 7316823796961456128. Current Snapshot: MvccSnapshot[applied={T|T < 7316823797164765184 or (T in {7316823797164765184})}]
I20260810 03:52:14.889822 16008 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:52:14.890027 16008 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:52:14.893597 16008 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 307cdcc454d4426f9c7730179a9d457f: Sending DeleteTablet for 3 replicas of tablet d73786d242ef42e488ea5f5d72731fc1
I20260810 03:52:14.894888 16531 tablet_service.cc:1514] Processing DeleteTablet for tablet d73786d242ef42e488ea5f5d72731fc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:52:14 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:56872
I20260810 03:52:14.894933 16789 tablet_service.cc:1514] Processing DeleteTablet for tablet d73786d242ef42e488ea5f5d72731fc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:52:14 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:52738
I20260810 03:52:14.895906 16260 tablet_service.cc:1514] Processing DeleteTablet for tablet d73786d242ef42e488ea5f5d72731fc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:52:14 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:38704
I20260810 03:52:14.896469 16986 tablet_replica.cc:330] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7: stopping tablet replica
I20260810 03:52:14.896786 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 16165
I20260810 03:52:14.896847 16986 raft_consensus.cc:2229] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:52:14.901015 16986 raft_consensus.cc:2258] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:52:14.907279 16986 ts_tablet_manager.cc:1905] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:52:14.908769 16987 tablet_replica.cc:330] T d73786d242ef42e488ea5f5d72731fc1 P 98a75889a2774502b1420456689c3856: stopping tablet replica
W20260810 03:52:14.916097 15970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.153.65:43015 (error 108)
I20260810 03:52:14.917764 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:14.917415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:43015 (local address 127.0.0.1:38714)
0810 03:52:14.917569 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 03:52:14.917660 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:43015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 03:52:14.917984 15970 catalog_manager.cc:4584] TS 291d41adc00445989379fd80ef909177 (127.27.153.65:43015): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d73786d242ef42e488ea5f5d72731fc1: Network error: Client connection negotiation failed: client connection to 127.27.153.65:43015: connect: Connection refused (error 111)
I20260810 03:52:14.925812 16986 ts_tablet_manager.cc:1918] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:52:14.925948 16986 log.cc:1196] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/d73786d242ef42e488ea5f5d72731fc1
I20260810 03:52:14.926283 16986 ts_tablet_manager.cc:1939] T d73786d242ef42e488ea5f5d72731fc1 P 9d78169e736f4316a4cb71f2b2f26df7: Deleting consensus metadata
I20260810 03:52:14.935807 16987 raft_consensus.cc:2229] T d73786d242ef42e488ea5f5d72731fc1 P 98a75889a2774502b1420456689c3856 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:52:14.941042 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 16417
I20260810 03:52:14.941462 16987 raft_consensus.cc:2258] T d73786d242ef42e488ea5f5d72731fc1 P 98a75889a2774502b1420456689c3856 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:52:14.956837 16987 ts_tablet_manager.cc:1905] T d73786d242ef42e488ea5f5d72731fc1 P 98a75889a2774502b1420456689c3856: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:52:14.970229 16987 ts_tablet_manager.cc:1918] T d73786d242ef42e488ea5f5d72731fc1 P 98a75889a2774502b1420456689c3856: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:52:14.970396 16987 log.cc:1196] T d73786d242ef42e488ea5f5d72731fc1 P 98a75889a2774502b1420456689c3856: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/d73786d242ef42e488ea5f5d72731fc1
I20260810 03:52:14.970724 16987 ts_tablet_manager.cc:1939] T d73786d242ef42e488ea5f5d72731fc1 P 98a75889a2774502b1420456689c3856: Deleting consensus metadata
I20260810 03:52:14.971807 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:14.971449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:43015 (local address 127.0.0.1:38718)
0810 03:52:14.971663 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 03:52:14.971718 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:43015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260810 03:52:14.972456 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:14.972250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:43015 (local address 127.0.0.1:38734)
0810 03:52:14.972361 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 03:52:14.972398 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:43015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 03:52:14.976064 15970 catalog_manager.cc:4841] TS 98a75889a2774502b1420456689c3856 (127.27.153.67:46857): tablet d73786d242ef42e488ea5f5d72731fc1 (table test-table [id=c30e4b535bbc4719aec077dc35066ada]) successfully deleted
I20260810 03:52:15.033522 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:15.033069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:43015 (local address 127.0.0.1:38750)
0810 03:52:15.033324 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 03:52:15.033405 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:43015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260810 03:52:15.034600 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:15.034269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:43015 (local address 127.0.0.1:38758)
0810 03:52:15.034447 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 03:52:15.034508 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:43015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260810 03:52:15.086045 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 16669
I20260810 03:52:15.086354 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:15.085982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:34059 (local address 127.0.0.1:56882)
0810 03:52:15.086201 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 03:52:15.086259 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.66:34059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260810 03:52:15.115340 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:15.114781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:43015 (local address 127.0.0.1:38770)
0810 03:52:15.115131 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0810 03:52:15.115219 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:43015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235}
I20260810 03:52:15.117868 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:52:15.117455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:43015 (local address 127.0.0.1:38776)
0810 03:52:15.117684 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 03:52:15.117758 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:43015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260810 03:52:15.134444 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 15943
I20260810 03:52:15.147374 28261 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 03:52:15.493402 28261 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 03:52:15.493573 28261 mini_ranger.cc:66] Stopping Ranger...
I20260810 03:52:15.849221 28261 mini_ranger.cc:68] Stopped Ranger
2026-08-10 03:52:15.849 UTC [15199] LOG: received smart shutdown request
2026-08-10 03:52:15.853 UTC [15199] LOG: background worker "logical replication launcher" (PID 15220) exited with exit code 1
2026-08-10 03:52:15.853 UTC [15210] LOG: shutting down
2026-08-10 03:52:16.017 UTC [15199] LOG: database system is shut down
2026-08-10T03:52:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51325 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:52:16 dist-test-slave-pww0 krb5kdc[17006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:52:16 dist-test-slave-pww0 krb5kdc[17006](info): set up 2 sockets
Aug 10 03:52:16 dist-test-slave-pww0 krb5kdc[17006](info): commencing operation
krb5kdc: starting...
W20260810 03:52:17.315390 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:52:17 dist-test-slave-pww0 krb5kdc[17006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333937, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:52:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:52:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.153.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.153.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.153.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/rangeradmin_127.27.153.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.153.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/rangeradmin_127.27.153.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.153.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.153.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.153.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/rangerlookup_127.27.153.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.153.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/rangerlookup_127.27.153.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.153.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.153.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.153.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/HTTP_127.27.153.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.153.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/HTTP_127.27.153.84@KRBTEST.COM.keytab.
I20260810 03:52:17.498512 28261 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskM5UkAJ/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/postgres -l logfile start
2026-08-10 03:52:19.964 UTC [17071] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 03:52:19.964 UTC [17071] LOG: listening on IPv4 address "127.27.153.84", port 47383
2026-08-10 03:52:19.968 UTC [17071] LOG: listening on Unix socket "/tmp/.s.PGSQL.47383"
2026-08-10 03:52:19.989 UTC [17075] LOG: database system was shut down at 2026-08-10 03:52:18 UTC
2026-08-10 03:52:19.993 UTC [17071] LOG: database system is ready to accept connections
I20260810 03:52:21.090868 28261 mini_postgres.cc:96] Postgres bound to 47383
2026-08-10 03:52:21.098 UTC [17086] FATAL: database "slave" does not exist
127.27.153.84:47383 - accepting connections
I20260810 03:52:21.099017 28261 mini_ranger.cc:161] Starting Ranger...
I20260810 03:52:21.115716 28261 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 03:52:21.582113 28261 mini_ranger.cc:87] Created ranger Postgres database
I20260810 03:52:21.582206 28261 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ranger-admin
2026-08-10 03:52:21,850 [I] DB FLAVOR :POSTGRES
2026-08-10 03:52:21,850 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 03:52:21,851 [I] Checking connection..
2026-08-10 03:52:21,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:47383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 03:52:22,105 [I] Checking connection passed.
2026-08-10 03:52:22,105 [I] --------- Verifying version history table ---------
2026-08-10 03:52:22,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:47383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:52:22,380 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 03:52:22,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:47383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:52:22,656 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 03:52:22,657 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 03:52:22,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:47383/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 03:52:22.900 UTC [17202] WARNING: there is no transaction in progress
2026-08-10 03:52:22,909 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 03:52:22,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:47383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 03:52:23,167 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 03:52:23,167 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 03:52:23,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:47383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 03:52:23,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:47383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 03:52:23,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:47383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-pww0.c.gcp-upstream.internal','N') ;"
2026-08-10 03:52:23,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:47383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 03:52:24,203 [I] Table x_portal_user does not exist in database ranger
2026-08-10 03:52:24,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:47383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 03:52:24,468 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 03:52:24,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:47383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 03:52:24,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:47383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 03:52:25,023 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 03:52:25,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:47383/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 03:52:26.032 UTC [17410] WARNING: there is no transaction in progress
2026-08-10 03:52:26.048 UTC [17410] WARNING: there is no transaction in progress
2026-08-10 03:52:26.065 UTC [17410] WARNING: there is no transaction in progress
2026-08-10 03:52:26.081 UTC [17410] WARNING: there is no transaction in progress
2026-08-10 03:52:26.096 UTC [17410] WARNING: there is no transaction in progress
2026-08-10 03:52:26.115 UTC [17410] WARNING: there is no transaction in progress
2026-08-10 03:52:26.177 UTC [17410] WARNING: there is no transaction in progress
2026-08-10 03:52:26.184 UTC [17410] WARNING: there is no transaction in progress
2026-08-10 03:52:26.198 UTC [17410] WARNING: there is no transaction in progress
2026-08-10 03:52:26.210 UTC [17410] WARNING: there is no transaction in progress
2026-08-10 03:52:26.221 UTC [17410] WARNING: there is no transaction in progress
2026-08-10 03:52:26.235 UTC [17410] WARNING: there is no transaction in progress
2026-08-10 03:52:26.246 UTC [17410] WARNING: there is no transaction in progress
2026-08-10 03:52:26.253 UTC [17410] WARNING: there is no transaction in progress
2026-08-10 03:52:26.259 UTC [17410] WARNING: there is no transaction in progress
2026-08-10 03:52:26.266 UTC [17410] WARNING: there is no transaction in progress
2026-08-10 03:52:26,943 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 03:52:26,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:47383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 03:52:27,215 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 03:52:27,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:47383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 03:52:27,492 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 03:52:27,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:47383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 03:52:27,773 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 03:52:27,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:47383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 03:52:28,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:47383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 03:52:28,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:47383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-pww0.c.gcp-upstream.internal';"
2026-08-10 03:52:28,551 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 03:52:28,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:47383/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 03:52:28,816 [I] DB_PATCHES have already been applied
I20260810 03:52:28.821792 28261 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ranger-admin:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskM5UkAJ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 03:52:28.821866 28261 mini_ranger.cc:193] Using host: 127.27.153.84
I20260810 03:52:28.826357 28261 mini_ranger.cc:227] Ranger admin URL: http://127.27.153.84:49363
Aug 10, 2026 3:52:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:52:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:52:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 3:52:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 3:52:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 3:52:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 3:52:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 03:52:30.157292 28261 mini_ranger.cc:160] Time spent starting Ranger: real 9.058s user 0.000s sys 0.012s
Aug 10, 2026 3:52:51 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 3:52:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 10, 2026 3:52:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 3:52:51 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 3:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 3:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 3:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 3:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 3:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 3:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 3:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 3:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 3:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 3:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 3:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 3:52:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 3:52:51 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 3:52:52 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 03:52:53.518356 28261 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.153.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.153.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.153.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/rangerkms_127.27.153.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.153.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/rangerkms_127.27.153.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.153.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.153.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.153.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/HTTP_127.27.153.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.153.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/HTTP_127.27.153.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 03:52:53 dist-test-slave-pww0 krb5kdc[17006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333973, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 03:52:53.612568 28261 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 03:52:53.624354 28261 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 03:52:54.522120 28261 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 03:52:54.522217 28261 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ranger-kms
I20260810 03:52:54.522239 28261 mini_ranger_kms.cc:227] Using postgres at 127.27.153.84:47383
2026-08-10 03:52:54,707 [I] DB FLAVOR :POSTGRES
2026-08-10 03:52:54,707 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 03:52:54,707 [I] Checking connection
2026-08-10 03:52:54,707 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:47383/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 03:52:54,994 [I] connection success
2026-08-10 03:52:54,994 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 03:52:54,994 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 03:52:54,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:47383/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 03:52:55,251 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 03:52:55,251 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 03:52:55,251 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 03:52:55,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.153.84:47383/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 03:52:55,547 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 03:52:56.135177 28261 mini_ranger_kms.cc:310] Created kms service
I20260810 03:52:56.321178 28261 mini_ranger_kms.cc:326] Created kudu user
I20260810 03:52:56.373703 28261 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 03:52:56 dist-test-slave-pww0 krb5kdc[17006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333973, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.153.84@KRBTEST.COM
I20260810 03:52:56.817485 28261 mini_ranger_kms.cc:384] Added ranger policy
I20260810 03:52:56.817652 28261 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ranger-kms:/tmp/dist-test-taskM5UkAJ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskM5UkAJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 03:52:56.817672 28261 mini_ranger_kms.cc:240] Using host: 127.27.153.84
I20260810 03:52:56.822598 28261 mini_ranger_kms.cc:276] Ranger KMS PID: 17732
I20260810 03:52:56.822714 28261 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.153.84:52679
Aug 10, 2026 3:52:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 3:52:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 3:52:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 3:52:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 3:52:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 3:52:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 03:52:59 dist-test-slave-pww0 krb5kdc[17006](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786333979, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.153.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:52:59.454172 28261 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.842s user 0.012s sys 0.160s
I20260810 03:52:59.454322 28261 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 03:52:59.454408 28261 mini_ranger_kms.cc:401] 127.27.153.84:52679/kms/v1/keys
Aug 10 03:52:59 dist-test-slave-pww0 krb5kdc[17006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333979, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.153.84@KRBTEST.COM for HTTP/127.27.153.84@KRBTEST.COM
Aug 10 03:53:00 dist-test-slave-pww0 krb5kdc[17006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333979, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.153.84@KRBTEST.COM for HTTP/127.27.153.84@KRBTEST.COM
Aug 10 03:53:02 dist-test-slave-pww0 krb5kdc[17006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:53:02.526995 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:39401
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:40597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:39401
--ranger_config_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.153.84:52679/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:53:02.645545 17817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:02.645872 17817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:02.645934 17817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:02.650231 17817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:53:02.650329 17817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:02.650357 17817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:02.650379 17817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:53:02.650408 17817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:53:02.656365 17817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.153.84:52679/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:39401
--ranger_config_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:39401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:02.657593 17817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:02.658727 17817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:02.668005 17826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:02.667965 17830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:02.667989 17825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:02.668460 17817 server_base.cc:1034] running on GCE node
I20260810 03:53:02.669098 17817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:02.670498 17817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:02.671681 17817 hybrid_clock.cc:648] HybridClock initialized: now 1786333982671656 us; error 52 us; skew 500 ppm
Aug 10 03:53:02 dist-test-slave-pww0 krb5kdc[17006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333982, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:02.677671 17817 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:53:02.678778 17817 webserver.cc:460] Webserver started at http://127.27.153.126:40013/ using document root <none> and password file <none>
I20260810 03:53:02.679358 17817 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:02.679421 17817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:02.679697 17817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:53:02 dist-test-slave-pww0 krb5kdc[17006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333982, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for HTTP/127.27.153.84@KRBTEST.COM
I20260810 03:53:02.740491 17817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "055d6244c60e417d9766c9889ca6c090"
format_stamp: "Formatted at 2026-08-10 03:53:02 on dist-test-slave-pww0"
server_key: "005bdbd8a0fecb91e9472dc5c6d747e9"
server_key_iv: "b0ba2240fde78e9e3b100c4a590f8d71"
server_key_version: "kuduclusterkey@0"
I20260810 03:53:02.741291 17817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "055d6244c60e417d9766c9889ca6c090"
format_stamp: "Formatted at 2026-08-10 03:53:02 on dist-test-slave-pww0"
server_key: "005bdbd8a0fecb91e9472dc5c6d747e9"
server_key_iv: "b0ba2240fde78e9e3b100c4a590f8d71"
server_key_version: "kuduclusterkey@0"
I20260810 03:53:02.746819 17817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 03:53:02.772010 17841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:02.773399 17817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 03:53:02.773523 17817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "055d6244c60e417d9766c9889ca6c090"
format_stamp: "Formatted at 2026-08-10 03:53:02 on dist-test-slave-pww0"
server_key: "005bdbd8a0fecb91e9472dc5c6d747e9"
server_key_iv: "b0ba2240fde78e9e3b100c4a590f8d71"
server_key_version: "kuduclusterkey@0"
I20260810 03:53:02.773643 17817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:02.811336 17817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:02.814401 17817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:02.814580 17817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:02.822014 17817 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:39401
I20260810 03:53:02.822047 17941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:39401 every 8 connection(s)
I20260810 03:53:02.823024 17817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:53:02.825954 17942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:53:02.827924 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 17817
I20260810 03:53:02.828024 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:53:02.828284 28261 external_mini_cluster.cc:1428] Setting key 2a71f1f28ad4e1bbc36d07efecfd6dc3
I20260810 03:53:02.832698 17942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090: Bootstrap starting.
I20260810 03:53:02.835361 17942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090: Neither blocks nor log segments found. Creating new log.
I20260810 03:53:02.836359 17942 log.cc:824] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:53:02 dist-test-slave-pww0 krb5kdc[17006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:53:02.838677 17942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090: No bootstrap required, opened a new log
I20260810 03:53:02.842862 17942 raft_consensus.cc:348] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055d6244c60e417d9766c9889ca6c090" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39401 } }
I20260810 03:53:02.843186 17942 raft_consensus.cc:374] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:53:02.843238 17942 raft_consensus.cc:729] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 055d6244c60e417d9766c9889ca6c090, State: Initialized, Role: FOLLOWER
I20260810 03:53:02.843771 17942 consensus_queue.cc:260] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055d6244c60e417d9766c9889ca6c090" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39401 } }
I20260810 03:53:02.843937 17942 raft_consensus.cc:388] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:53:02.844005 17942 raft_consensus.cc:482] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:53:02.844079 17947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:02.829631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41646 (local address 127.27.153.126:39401)
0810 03:53:02.830050 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:02.830066 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:02.830104 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:53:02.830786 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:02.830804 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:02.831162 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:02.831474 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:02.832410 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:02.833430 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:02.834771 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:02.835294 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:02.835427 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:02.838203 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:02.838211 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:02.838229 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:02.838260 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:02.840877 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:02.841341 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:02.841348 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:02.841355 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:02.841495 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:02.841750 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:02.841754 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:02.841756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:02.842340 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:02.842557 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:02.842722 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 03:53:02.843018 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":122,"threads_started":1}
I20260810 03:53:02.844116 17942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:53:02.845224 17942 raft_consensus.cc:504] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055d6244c60e417d9766c9889ca6c090" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39401 } }
I20260810 03:53:02.845575 17942 leader_election.cc:302] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 055d6244c60e417d9766c9889ca6c090; no voters:
I20260810 03:53:02.845921 17942 leader_election.cc:288] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:53:02.846031 17949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:53:02.846279 17949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [term 1 LEADER]: Becoming Leader. State: Replica: 055d6244c60e417d9766c9889ca6c090, State: Running, Role: LEADER
I20260810 03:53:02.846598 17949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055d6244c60e417d9766c9889ca6c090" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39401 } }
I20260810 03:53:02.847205 17942 sys_catalog.cc:564] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:53:02.848470 17950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "055d6244c60e417d9766c9889ca6c090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055d6244c60e417d9766c9889ca6c090" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39401 } } }
I20260810 03:53:02.848611 17950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [sys.catalog]: This master's current role is: LEADER
I20260810 03:53:02.848999 17965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:53:02.849030 17951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 055d6244c60e417d9766c9889ca6c090. Latest consensus state: current_term: 1 leader_uuid: "055d6244c60e417d9766c9889ca6c090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055d6244c60e417d9766c9889ca6c090" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39401 } } }
I20260810 03:53:02.849192 17951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [sys.catalog]: This master's current role is: LEADER
I20260810 03:53:02.852515 17965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:53:02.853138 17942 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 03:53:02.863348 17965 catalog_manager.cc:1347] Generated new cluster ID: bbd848f9e20747deb545eeebc3ae98ae
I20260810 03:53:02.863530 17965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:53:02.883396 17965 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:53:02.884968 17965 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:53:02.894042 17965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090: Generated new TSK 0
I20260810 03:53:02.895040 17965 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.153.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
I20260810 03:53:03.592712 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.153.84:52679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:39401
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:40597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:53:03.765301 18021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:03.765688 18021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:03.765756 18021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:03.772126 18021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:03.772215 18021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:03.772305 18021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:53:03.780654 18021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.153.84:52679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:39401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:03.782262 18021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:03.783797 18021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:03.799144 18030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:03.799268 18029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:03.799448 18033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:03.800850 18021 server_base.cc:1034] running on GCE node
I20260810 03:53:03.801450 18021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:03.802354 18021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:03.803558 18021 hybrid_clock.cc:648] HybridClock initialized: now 1786333983803525 us; error 60 us; skew 500 ppm
Aug 10 03:53:03 dist-test-slave-pww0 krb5kdc[17006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333983, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:03.806816 18021 init.cc:377] Logged in from keytab as kudu/127.27.153.65@KRBTEST.COM (short username kudu)
I20260810 03:53:03.808166 18021 webserver.cc:460] Webserver started at http://127.27.153.65:37685/ using document root <none> and password file <none>
I20260810 03:53:03.808966 18021 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:03.809077 18021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:03.809369 18021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:53:03 dist-test-slave-pww0 krb5kdc[17006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333983, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for HTTP/127.27.153.84@KRBTEST.COM
I20260810 03:53:03.831645 18021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "37d8b96b9dc944f999a231f961b5fa76"
format_stamp: "Formatted at 2026-08-10 03:53:03 on dist-test-slave-pww0"
server_key: "9749ac822ac8ad782dad07944666a61a"
server_key_iv: "729282645e967d346b0601da942fbbbb"
server_key_version: "kuduclusterkey@0"
I20260810 03:53:03.832720 18021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "37d8b96b9dc944f999a231f961b5fa76"
format_stamp: "Formatted at 2026-08-10 03:53:03 on dist-test-slave-pww0"
server_key: "9749ac822ac8ad782dad07944666a61a"
server_key_iv: "729282645e967d346b0601da942fbbbb"
server_key_version: "kuduclusterkey@0"
I20260810 03:53:03.841037 18021 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260810 03:53:03.862512 18044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:03.864707 18021 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260810 03:53:03.864899 18021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "37d8b96b9dc944f999a231f961b5fa76"
format_stamp: "Formatted at 2026-08-10 03:53:03 on dist-test-slave-pww0"
server_key: "9749ac822ac8ad782dad07944666a61a"
server_key_iv: "729282645e967d346b0601da942fbbbb"
server_key_version: "kuduclusterkey@0"
I20260810 03:53:03.865088 18021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:03.894145 18021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:03.898219 18021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:03.898509 18021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:03.899397 18021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:53:03.900633 18021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:53:03.900748 18021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:03.900879 18021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:53:03.900966 18021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:03.916266 18021 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:44597
I20260810 03:53:03.917589 18021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:53:03.919406 18234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:44597 every 8 connection(s)
I20260810 03:53:03.923761 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 18021
I20260810 03:53:03.923911 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:53:03.924256 28261 external_mini_cluster.cc:1428] Setting key bd6386a800e2875207872dbe6c4c8c30
Aug 10 03:53:03 dist-test-slave-pww0 krb5kdc[17006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333983, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:53:03.937469 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:03.921429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:53505 (local address 127.27.153.126:39401)
0810 03:53:03.921733 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:03.921738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:03.922851 (+ 1113us) server_negotiation.cc:407] Connection header received
0810 03:53:03.924207 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:03.924212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:03.924275 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:03.924341 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:03.926062 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:03.926940 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:03.928708 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:03.929742 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:03.929829 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:03.932845 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:03.932849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:03.932851 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:03.932880 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:03.934785 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:03.935444 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:03.935447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:03.935450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:03.935577 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:03.936028 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:03.936031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:03.936033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:03.936286 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:03.936372 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:03.937124 (+ 752us) server_negotiation.cc:299] Negotiation successful
0810 03:53:03.937271 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":100,"threads_started":1}
I20260810 03:53:03.939559 18240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:03.922017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39401 (local address 127.27.153.65:53505)
0810 03:53:03.922654 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:03.922730 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:03.923937 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:03.924803 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:03.924813 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:03.925192 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:03.925885 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:03.925905 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:03.927104 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:03.927107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:03.928549 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:03.928563 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:03.930067 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:03.930071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:03.930319 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:03.931057 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:03.931075 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:03.932622 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:03.934932 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:03.934939 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:03.934961 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:03.935327 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:03.935703 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:03.935705 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:03.935707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:03.935911 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:03.936445 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:03.936453 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:03.936825 (+ 372us) client_negotiation.cc:769] Sending connection context
0810 03:53:03.937980 (+ 1155us) client_negotiation.cc:240] Negotiation successful
0810 03:53:03.938369 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":177,"threads_started":1}
I20260810 03:53:03.941733 18235 heartbeater.cc:347] Connected to a master server at 127.27.153.126:39401
I20260810 03:53:03.942094 18235 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:03.942821 18235 heartbeater.cc:511] Master 127.27.153.126:39401 requested a full tablet report, sending...
I20260810 03:53:03.944830 17876 ts_manager.cc:194] Registered new tserver with Master: 37d8b96b9dc944f999a231f961b5fa76 (127.27.153.65:44597)
I20260810 03:53:03.946442 17876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:53505
WARNING: no policy specified for kudu/127.27.153.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
I20260810 03:53:03.986831 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.153.84:52679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:39401
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:40597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:53:04 dist-test-slave-pww0 krb5kdc[17006](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786333984, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 03:53:04.165091 18249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:04.165599 18249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:04.165829 18249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:04.172400 18249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:04.172566 18249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:04.172700 18249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:53:04.181293 18249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.153.84:52679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:39401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:04.183163 18249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:04.184775 18249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:04.202217 18260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:04.202219 18263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:04.203377 18259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:04.204089 18249 server_base.cc:1034] running on GCE node
I20260810 03:53:04.204731 18249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:04.205637 18249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:04.208870 18249 hybrid_clock.cc:648] HybridClock initialized: now 1786333984208733 us; error 147 us; skew 500 ppm
Aug 10 03:53:04 dist-test-slave-pww0 krb5kdc[17006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333984, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:04.213099 18249 init.cc:377] Logged in from keytab as kudu/127.27.153.66@KRBTEST.COM (short username kudu)
I20260810 03:53:04.214537 18249 webserver.cc:460] Webserver started at http://127.27.153.66:37709/ using document root <none> and password file <none>
I20260810 03:53:04.215420 18249 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:04.215613 18249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:04.215957 18249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:53:04 dist-test-slave-pww0 krb5kdc[17006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333984, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for HTTP/127.27.153.84@KRBTEST.COM
I20260810 03:53:04.241230 18249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "11a8abf9e1554d5ba8b4cf6c6463b3b0"
format_stamp: "Formatted at 2026-08-10 03:53:04 on dist-test-slave-pww0"
server_key: "0b30788306f3c2af840985802b46d884"
server_key_iv: "b062af49a503c96e826e50672b170938"
server_key_version: "kuduclusterkey@0"
I20260810 03:53:04.242290 18249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "11a8abf9e1554d5ba8b4cf6c6463b3b0"
format_stamp: "Formatted at 2026-08-10 03:53:04 on dist-test-slave-pww0"
server_key: "0b30788306f3c2af840985802b46d884"
server_key_iv: "b062af49a503c96e826e50672b170938"
server_key_version: "kuduclusterkey@0"
I20260810 03:53:04.251413 18249 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.000s
I20260810 03:53:04.287878 18271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:04.289959 18249 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.003s sys 0.000s
I20260810 03:53:04.290251 18249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "11a8abf9e1554d5ba8b4cf6c6463b3b0"
format_stamp: "Formatted at 2026-08-10 03:53:04 on dist-test-slave-pww0"
server_key: "0b30788306f3c2af840985802b46d884"
server_key_iv: "b062af49a503c96e826e50672b170938"
server_key_version: "kuduclusterkey@0"
I20260810 03:53:04.290606 18249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:04.310182 18249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:04.314374 18249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:04.314667 18249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:04.315572 18249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:53:04.316893 18249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:53:04.317019 18249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 03:53:04.317127 18249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:53:04.317190 18249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:04.330834 18249 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:35343
I20260810 03:53:04.332221 18249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:53:04.333210 18452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:35343 every 8 connection(s)
I20260810 03:53:04.338546 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 18249
I20260810 03:53:04.338676 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:53:04.339025 28261 external_mini_cluster.cc:1428] Setting key 211a52a92cd9e885ae23afaa016cf2ae
Aug 10 03:53:04 dist-test-slave-pww0 krb5kdc[17006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333984, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:53:04.353117 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:04.334467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:49391 (local address 127.27.153.126:39401)
0810 03:53:04.336251 (+ 1784us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:04.336257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:04.336277 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:53:04.337128 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:04.337132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:04.337197 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:04.337303 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:04.338822 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:04.339745 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:04.344235 (+ 4490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:04.344588 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:04.344673 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:04.348224 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:04.348229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:04.348232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:04.348266 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:04.350121 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:04.350895 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:04.350900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:04.350902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:04.350996 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:04.351500 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:04.351505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:04.351507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:04.351791 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:04.351909 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:04.352739 (+ 830us) server_negotiation.cc:299] Negotiation successful
0810 03:53:04.352908 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1674}
WARNING: no policy specified for kudu/127.27.153.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.67@KRBTEST.COM" created.
I20260810 03:53:04.375571 18459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:04.335025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39401 (local address 127.27.153.66:49391)
0810 03:53:04.335705 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:04.335769 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:04.336830 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:04.337514 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:04.337523 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:04.337900 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:04.338615 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:04.338637 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:04.339949 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:04.339953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:04.344008 (+ 4055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:04.344033 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:04.344923 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:04.344926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:04.345155 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:04.345929 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:04.345952 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:04.347942 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:04.350337 (+ 2395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:04.350345 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:04.350368 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:04.350763 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:04.351104 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:04.351107 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:04.351108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:04.351319 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:04.351980 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:04.351989 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:04.352418 (+ 429us) client_negotiation.cc:769] Sending connection context
0810 03:53:04.361683 (+ 9265us) client_negotiation.cc:240] Negotiation successful
0810 03:53:04.374199 (+ 12516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":180,"threads_started":1}
I20260810 03:53:04.377303 18456 heartbeater.cc:347] Connected to a master server at 127.27.153.126:39401
I20260810 03:53:04.377593 18456 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:04.378268 18456 heartbeater.cc:511] Master 127.27.153.126:39401 requested a full tablet report, sending...
I20260810 03:53:04.379813 17876 ts_manager.cc:194] Registered new tserver with Master: 11a8abf9e1554d5ba8b4cf6c6463b3b0 (127.27.153.66:35343)
I20260810 03:53:04.380591 17876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:49391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
I20260810 03:53:04.388257 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.153.84:52679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:39401
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:40597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:53:04.559038 18468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:04.559527 18468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:04.559648 18468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:04.565904 18468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:04.566066 18468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:04.566197 18468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:53:04.574131 18468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.153.84:52679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:39401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:04.575827 18468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:04.577260 18468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Aug 10 03:53:04 dist-test-slave-pww0 krb5kdc[17006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333984, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for HTTP/127.27.153.84@KRBTEST.COM
W20260810 03:53:04.591790 18479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:04.592900 18468 server_base.cc:1034] running on GCE node
W20260810 03:53:04.593199 18485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:04.598553 18477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:04.599395 18468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:04.600373 18468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:04.601676 18468 hybrid_clock.cc:648] HybridClock initialized: now 1786333984601613 us; error 89 us; skew 500 ppm
Aug 10 03:53:04 dist-test-slave-pww0 krb5kdc[17006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333984, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:04.605661 18468 init.cc:377] Logged in from keytab as kudu/127.27.153.67@KRBTEST.COM (short username kudu)
I20260810 03:53:04.606949 18468 webserver.cc:460] Webserver started at http://127.27.153.67:42389/ using document root <none> and password file <none>
I20260810 03:53:04.607841 18468 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:04.607971 18468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:04.608263 18468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 03:53:04 dist-test-slave-pww0 krb5kdc[17006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333984, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for HTTP/127.27.153.84@KRBTEST.COM
I20260810 03:53:04.625694 18468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "4a040d1ed5ee47deb9d2fba42d885bd6"
format_stamp: "Formatted at 2026-08-10 03:53:04 on dist-test-slave-pww0"
server_key: "602401209422bda1b048ea11f590606b"
server_key_iv: "42c5cf091bc00ed7b31bc005771740c1"
server_key_version: "kuduclusterkey@0"
I20260810 03:53:04.626907 18468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "4a040d1ed5ee47deb9d2fba42d885bd6"
format_stamp: "Formatted at 2026-08-10 03:53:04 on dist-test-slave-pww0"
server_key: "602401209422bda1b048ea11f590606b"
server_key_iv: "42c5cf091bc00ed7b31bc005771740c1"
server_key_version: "kuduclusterkey@0"
I20260810 03:53:04.643304 18468 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.004s sys 0.000s
I20260810 03:53:04.667181 18501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:04.669065 18468 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.003s sys 0.000s
I20260810 03:53:04.669214 18468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "4a040d1ed5ee47deb9d2fba42d885bd6"
format_stamp: "Formatted at 2026-08-10 03:53:04 on dist-test-slave-pww0"
server_key: "602401209422bda1b048ea11f590606b"
server_key_iv: "42c5cf091bc00ed7b31bc005771740c1"
server_key_version: "kuduclusterkey@0"
I20260810 03:53:04.669369 18468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:04.694060 18468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:04.697981 18468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:04.698192 18468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:04.699014 18468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:53:04.700198 18468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:53:04.700269 18468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:04.700393 18468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:53:04.700435 18468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:04.711301 18468 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:36685
I20260810 03:53:04.711341 18703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:36685 every 8 connection(s)
I20260810 03:53:04.712386 18468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:53:04.715921 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 18468
I20260810 03:53:04.716048 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:53:04.716344 28261 external_mini_cluster.cc:1428] Setting key 4a0e2b0abe08978b9a62c03bdfba4a41
Aug 10 03:53:04 dist-test-slave-pww0 krb5kdc[17006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333984, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:53:04.727634 18239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:04.714540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:38979 (local address 127.27.153.126:39401)
0810 03:53:04.714679 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:04.714684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:04.715736 (+ 1052us) server_negotiation.cc:407] Connection header received
0810 03:53:04.716914 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:04.716918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:04.716974 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:04.717028 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:04.719285 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:04.720053 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:04.721218 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:04.721454 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:04.721545 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:04.723962 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:04.723965 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:04.723967 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:04.723995 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:04.725455 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:04.725994 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:04.725997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:04.725999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:04.726084 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:04.726430 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:04.726433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:04.726435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:04.726640 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:04.726730 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:04.727294 (+ 564us) server_negotiation.cc:299] Negotiation successful
0810 03:53:04.727427 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:53:04.728605 18706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:04.715012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39401 (local address 127.27.153.67:38979)
0810 03:53:04.715564 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:04.715622 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:04.716636 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:04.717568 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:04.717574 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:04.718556 (+ 982us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:04.719139 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:04.719155 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:04.720181 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:04.720185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:04.721097 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:04.721108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:04.721600 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:04.721603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:04.721789 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:04.722374 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:04.722392 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:04.723788 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:04.725576 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:04.725583 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:04.725602 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:04.725894 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:04.726169 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:04.726172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:04.726175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:04.726341 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:04.726751 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:04.726758 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:04.727041 (+ 283us) client_negotiation.cc:769] Sending connection context
0810 03:53:04.727223 (+ 182us) client_negotiation.cc:240] Negotiation successful
0810 03:53:04.727621 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":131,"threads_started":1}
I20260810 03:53:04.729934 18704 heartbeater.cc:347] Connected to a master server at 127.27.153.126:39401
I20260810 03:53:04.730186 18704 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:04.730729 18704 heartbeater.cc:511] Master 127.27.153.126:39401 requested a full tablet report, sending...
I20260810 03:53:04.731890 17874 ts_manager.cc:194] Registered new tserver with Master: 4a040d1ed5ee47deb9d2fba42d885bd6 (127.27.153.67:36685)
I20260810 03:53:04.732688 17874 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:38979
I20260810 03:53:04.743870 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:53:04.744563 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 18021
I20260810 03:53:04.768110 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 18249
I20260810 03:53:04.781211 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 18468
I20260810 03:53:04.800056 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 17817
I20260810 03:53:04.821398 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:39401
--webserver_interface=127.27.153.126
--webserver_port=40013
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:40597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:39401
--ranger_config_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.153.84:52679/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.153.84:52679/kms with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:53:04.936967 18707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:04.937245 18707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:04.937284 18707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:04.941407 18707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:53:04.941481 18707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:04.941500 18707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:04.941515 18707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:53:04.941530 18707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:53:04.947194 18707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.153.84:52679/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:39401
--ranger_config_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:39401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=40013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:04.948370 18707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:04.949440 18707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:04.958582 18721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:04.958587 18725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:04.958587 18722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:04.959259 18707 server_base.cc:1034] running on GCE node
I20260810 03:53:04.959883 18707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:04.961061 18707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:04.962216 18707 hybrid_clock.cc:648] HybridClock initialized: now 1786333984962191 us; error 47 us; skew 500 ppm
Aug 10 03:53:04 dist-test-slave-pww0 krb5kdc[17006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333984, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:04.965027 18707 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:53:04.966022 18707 webserver.cc:460] Webserver started at http://127.27.153.126:40013/ using document root <none> and password file <none>
I20260810 03:53:04.966564 18707 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:04.966647 18707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 03:53:05 dist-test-slave-pww0 krb5kdc[17006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333984, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for HTTP/127.27.153.84@KRBTEST.COM
I20260810 03:53:05.048313 18707 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:53:05.050954 18737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:05.052467 18707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:53:05.052611 18707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "055d6244c60e417d9766c9889ca6c090"
format_stamp: "Formatted at 2026-08-10 03:53:02 on dist-test-slave-pww0"
server_key: "005bdbd8a0fecb91e9472dc5c6d747e9"
server_key_iv: "b0ba2240fde78e9e3b100c4a590f8d71"
server_key_version: "kuduclusterkey@0"
I20260810 03:53:05.053212 18707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:05.073410 18707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:05.076483 18707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:05.076681 18707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:05.084022 18707 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:39401
I20260810 03:53:05.084105 18835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:39401 every 8 connection(s)
I20260810 03:53:05.085153 18707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:53:05.089524 18837 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:53:05.090448 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 18707
I20260810 03:53:05.090854 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.153.84:52679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:44597
--local_ip_for_outbound_sockets=127.27.153.65
--tserver_master_addrs=127.27.153.126:39401
--webserver_port=37685
--webserver_interface=127.27.153.65
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:40597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.153.84:52679/kms with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:53:05.091359 18837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090: Bootstrap starting.
I20260810 03:53:05.108409 18837 log.cc:824] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:05.115509 18837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:53:05.116003 18837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090: Bootstrap complete.
I20260810 03:53:05.120182 18837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055d6244c60e417d9766c9889ca6c090" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39401 } }
I20260810 03:53:05.120621 18837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 055d6244c60e417d9766c9889ca6c090, State: Initialized, Role: FOLLOWER
I20260810 03:53:05.121145 18837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055d6244c60e417d9766c9889ca6c090" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39401 } }
I20260810 03:53:05.121280 18837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:53:05.121352 18837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:53:05.121452 18837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:53:05.123660 18837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055d6244c60e417d9766c9889ca6c090" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39401 } }
I20260810 03:53:05.124054 18837 leader_election.cc:302] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 055d6244c60e417d9766c9889ca6c090; no voters:
I20260810 03:53:05.124394 18837 leader_election.cc:288] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 03:53:05.124571 18843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 03:53:05.124811 18843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [term 2 LEADER]: Becoming Leader. State: Replica: 055d6244c60e417d9766c9889ca6c090, State: Running, Role: LEADER
I20260810 03:53:05.125140 18843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055d6244c60e417d9766c9889ca6c090" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39401 } }
I20260810 03:53:05.125865 18837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:53:05.126230 18843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "055d6244c60e417d9766c9889ca6c090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055d6244c60e417d9766c9889ca6c090" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39401 } } }
I20260810 03:53:05.126355 18843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [sys.catalog]: This master's current role is: LEADER
I20260810 03:53:05.126575 18843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 055d6244c60e417d9766c9889ca6c090. Latest consensus state: current_term: 2 leader_uuid: "055d6244c60e417d9766c9889ca6c090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "055d6244c60e417d9766c9889ca6c090" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39401 } } }
I20260810 03:53:05.126639 18843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090 [sys.catalog]: This master's current role is: LEADER
I20260810 03:53:05.127163 18850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:53:05.130759 18850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:53:05.131434 18850 catalog_manager.cc:1259] Loaded cluster ID: bbd848f9e20747deb545eeebc3ae98ae
I20260810 03:53:05.131520 18850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:53:05.132012 18837 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 03:53:05.133889 18850 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:53:05.134588 18850 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090: Loaded TSK: 0
I20260810 03:53:05.135906 18850 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 03:53:05.220712 18841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:05.220991 18841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:05.221052 18841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:05.225682 18841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:05.225759 18841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:05.225843 18841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:53:05.232199 18841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.153.84:52679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:44597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=37685
--tserver_master_addrs=127.27.153.126:39401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:05.233371 18841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:05.234511 18841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:05.244773 18899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:05.245235 18905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:05.245589 18903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:05.245985 18841 server_base.cc:1034] running on GCE node
I20260810 03:53:05.246694 18841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:05.247583 18841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:05.251312 18841 hybrid_clock.cc:648] HybridClock initialized: now 1786333985251219 us; error 107 us; skew 500 ppm
Aug 10 03:53:05 dist-test-slave-pww0 krb5kdc[17006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333985, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:05.254532 18841 init.cc:377] Logged in from keytab as kudu/127.27.153.65@KRBTEST.COM (short username kudu)
I20260810 03:53:05.255757 18841 webserver.cc:460] Webserver started at http://127.27.153.65:37685/ using document root <none> and password file <none>
I20260810 03:53:05.256515 18841 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:05.256652 18841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 03:53:05 dist-test-slave-pww0 krb5kdc[17006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333985, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for HTTP/127.27.153.84@KRBTEST.COM
I20260810 03:53:05.286836 18841 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260810 03:53:05.290663 18923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:05.292495 18841 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 03:53:05.292685 18841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "37d8b96b9dc944f999a231f961b5fa76"
format_stamp: "Formatted at 2026-08-10 03:53:03 on dist-test-slave-pww0"
server_key: "9749ac822ac8ad782dad07944666a61a"
server_key_iv: "729282645e967d346b0601da942fbbbb"
server_key_version: "kuduclusterkey@0"
I20260810 03:53:05.293440 18841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:05.308574 18841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:05.312891 18841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:05.313112 18841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:05.313935 18841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:53:05.315092 18841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:53:05.315160 18841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:05.315219 18841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:53:05.315250 18841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:05.327500 18841 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:44597
I20260810 03:53:05.329012 18841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:53:05.330387 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 18841
I20260810 03:53:05.330884 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.153.84:52679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:35343
--local_ip_for_outbound_sockets=127.27.153.66
--tserver_master_addrs=127.27.153.126:39401
--webserver_port=37709
--webserver_interface=127.27.153.66
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:40597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.153.84:52679/kms with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:53:05.331923 19119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:44597 every 8 connection(s)
Aug 10 03:53:05 dist-test-slave-pww0 krb5kdc[17006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333985, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:53:05.361119 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:05.334662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39401 (local address 127.27.153.65:58103)
0810 03:53:05.338456 (+ 3794us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:05.338531 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:05.339543 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:05.341492 (+ 1949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:05.341503 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:05.341885 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:05.342567 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:05.342589 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:05.348132 (+ 5543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:05.348137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:05.349587 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:05.349609 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:05.351000 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:05.351004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:05.351209 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:05.352028 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:05.352052 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:05.353869 (+ 1817us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:05.356800 (+ 2931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:05.356808 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:05.356826 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:05.357242 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:05.357700 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:05.357703 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:05.357705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:05.358003 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:05.358902 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:05.358910 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:05.359273 (+ 363us) client_negotiation.cc:769] Sending connection context
0810 03:53:05.359583 (+ 310us) client_negotiation.cc:240] Negotiation successful
0810 03:53:05.359978 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3523,"thread_start_us":181,"threads_started":1}
I20260810 03:53:05.363263 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:05.334455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:58103 (local address 127.27.153.126:39401)
0810 03:53:05.338030 (+ 3575us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:05.338056 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:05.339560 (+ 1504us) server_negotiation.cc:407] Connection header received
0810 03:53:05.340059 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:05.340078 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:05.340499 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:05.340896 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:05.342782 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:05.347857 (+ 5075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:05.349798 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:05.350571 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:05.350755 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:05.354200 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:05.354209 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:05.354231 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:05.354274 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:05.356533 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:05.357411 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:05.357418 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:05.357424 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:05.357586 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:05.358138 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:05.358142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:05.358143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:05.358702 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:05.358840 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:05.361844 (+ 3004us) server_negotiation.cc:299] Negotiation successful
0810 03:53:05.362199 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3363,"thread_start_us":137,"threads_started":1}
I20260810 03:53:05.366204 19121 heartbeater.cc:347] Connected to a master server at 127.27.153.126:39401
I20260810 03:53:05.366499 19121 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:05.367187 19121 heartbeater.cc:511] Master 127.27.153.126:39401 requested a full tablet report, sending...
W20260810 03:53:05.369796 19121 heartbeater.cc:644] Failed to heartbeat to 127.27.153.126:39401 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 03:53:05.369951 19121 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:05.370191 19121 heartbeater.cc:511] Master 127.27.153.126:39401 requested a full tablet report, sending...
I20260810 03:53:05.371181 19121 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:05.371369 19121 heartbeater.cc:511] Master 127.27.153.126:39401 requested a full tablet report, sending...
W20260810 03:53:05.372189 19121 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:53:05.514919 19126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:05.515352 19126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:05.515434 19126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:05.521697 19126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:05.521795 19126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:05.521888 19126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:53:05.530421 19126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.153.84:52679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:35343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=37709
--tserver_master_addrs=127.27.153.126:39401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:05.532128 19126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:05.533725 19126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:05.549263 19141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:05.549280 19144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:05.549903 19140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:05.550287 19126 server_base.cc:1034] running on GCE node
I20260810 03:53:05.551002 19126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:05.551971 19126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:05.553185 19126 hybrid_clock.cc:648] HybridClock initialized: now 1786333985553142 us; error 52 us; skew 500 ppm
Aug 10 03:53:05 dist-test-slave-pww0 krb5kdc[17006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333985, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:05.557294 19126 init.cc:377] Logged in from keytab as kudu/127.27.153.66@KRBTEST.COM (short username kudu)
I20260810 03:53:05.558655 19126 webserver.cc:460] Webserver started at http://127.27.153.66:37709/ using document root <none> and password file <none>
I20260810 03:53:05.559510 19126 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:05.559643 19126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 03:53:05 dist-test-slave-pww0 krb5kdc[17006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333985, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for HTTP/127.27.153.84@KRBTEST.COM
I20260810 03:53:05.582109 19126 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260810 03:53:05.588487 19154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:05.590504 19126 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260810 03:53:05.590687 19126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "11a8abf9e1554d5ba8b4cf6c6463b3b0"
format_stamp: "Formatted at 2026-08-10 03:53:04 on dist-test-slave-pww0"
server_key: "0b30788306f3c2af840985802b46d884"
server_key_iv: "b062af49a503c96e826e50672b170938"
server_key_version: "kuduclusterkey@0"
I20260810 03:53:05.591710 19126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:05.641964 19126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:05.648128 19126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:05.648361 19126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:05.649103 19126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:53:05.650243 19126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:53:05.650324 19126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:05.650385 19126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:53:05.650449 19126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260810 03:53:05.662843 19126 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:35343
I20260810 03:53:05.662962 19359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:35343 every 8 connection(s)
I20260810 03:53:05.664393 19126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:53:05.670982 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 19126
I20260810 03:53:05.671423 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.153.84:52679/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:36685
--local_ip_for_outbound_sockets=127.27.153.67
--tserver_master_addrs=127.27.153.126:39401
--webserver_port=42389
--webserver_interface=127.27.153.67
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:40597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.153.84:52679/kms with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:53:05 dist-test-slave-pww0 krb5kdc[17006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333985, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:53:05.702200 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:05.667352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39401 (local address 127.27.153.66:50681)
0810 03:53:05.675892 (+ 8540us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:05.675986 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:05.677289 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:05.688254 (+ 10965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:05.688265 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:05.688672 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:05.689408 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:05.689429 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:05.690761 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:05.690765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:05.692276 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:05.692292 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:05.693111 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:05.693115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:05.693331 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:05.694065 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:05.694088 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:05.696150 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:05.698402 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:05.698413 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:05.698430 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:05.698816 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:05.699178 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:05.699181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:05.699183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:05.699390 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:05.699949 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:05.699956 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:05.700636 (+ 680us) client_negotiation.cc:769] Sending connection context
0810 03:53:05.700839 (+ 203us) client_negotiation.cc:240] Negotiation successful
0810 03:53:05.701190 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8205,"thread_start_us":155,"threads_started":1}
I20260810 03:53:05.703008 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:05.670054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:50681 (local address 127.27.153.126:39401)
0810 03:53:05.670215 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:05.670220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:05.676142 (+ 5922us) server_negotiation.cc:407] Connection header received
0810 03:53:05.687663 (+ 11521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:05.687667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:05.687760 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:05.687860 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:05.689619 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:05.690510 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:05.692443 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:05.692805 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:05.692908 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:05.696376 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:05.696381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:05.696384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:05.696416 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:05.698120 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:05.698947 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:05.698951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:05.698953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:05.699056 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:05.699529 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:05.699533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:05.699534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:05.699799 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:05.699892 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:05.702653 (+ 2761us) server_negotiation.cc:299] Negotiation successful
0810 03:53:05.702817 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 03:53:05.704833 19362 heartbeater.cc:347] Connected to a master server at 127.27.153.126:39401
I20260810 03:53:05.705132 19362 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:05.705796 19362 heartbeater.cc:511] Master 127.27.153.126:39401 requested a full tablet report, sending...
W20260810 03:53:05.707924 19362 heartbeater.cc:644] Failed to heartbeat to 127.27.153.126:39401 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 03:53:05.708052 19362 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:05.708264 19362 heartbeater.cc:511] Master 127.27.153.126:39401 requested a full tablet report, sending...
I20260810 03:53:05.709260 19362 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:05.709442 19362 heartbeater.cc:511] Master 127.27.153.126:39401 requested a full tablet report, sending...
W20260810 03:53:05.710250 19362 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:53:05.828446 19372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:05.828769 19372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:05.828842 19372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:05.834425 19372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:05.834630 19372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:05.834801 19372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:53:05.841563 19372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.153.84:52679/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:36685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=42389
--tserver_master_addrs=127.27.153.126:39401
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:05.842873 19372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:05.844090 19372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:05.856470 19389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:05.857065 19382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:05.857196 19381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:05.857245 19372 server_base.cc:1034] running on GCE node
I20260810 03:53:05.857933 19372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:05.858876 19372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:05.860095 19372 hybrid_clock.cc:648] HybridClock initialized: now 1786333985860037 us; error 78 us; skew 500 ppm
Aug 10 03:53:05 dist-test-slave-pww0 krb5kdc[17006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333985, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:05.863368 19372 init.cc:377] Logged in from keytab as kudu/127.27.153.67@KRBTEST.COM (short username kudu)
I20260810 03:53:05.864472 19372 webserver.cc:460] Webserver started at http://127.27.153.67:42389/ using document root <none> and password file <none>
I20260810 03:53:05.865094 19372 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:05.865190 19372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 03:53:05 dist-test-slave-pww0 krb5kdc[17006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333985, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for HTTP/127.27.153.84@KRBTEST.COM
I20260810 03:53:05.894730 19372 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:53:05.897782 19406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:05.900943 19372 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260810 03:53:05.901185 19372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "4a040d1ed5ee47deb9d2fba42d885bd6"
format_stamp: "Formatted at 2026-08-10 03:53:04 on dist-test-slave-pww0"
server_key: "602401209422bda1b048ea11f590606b"
server_key_iv: "42c5cf091bc00ed7b31bc005771740c1"
server_key_version: "kuduclusterkey@0"
I20260810 03:53:05.902041 19372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:05.934386 19372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:05.938472 19372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:05.938771 19372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:05.939633 19372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:53:05.940902 19372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:53:05.941017 19372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:05.941134 19372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:53:05.941200 19372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:05.952287 19372 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:36685
I20260810 03:53:05.952358 19605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:36685 every 8 connection(s)
I20260810 03:53:05.953466 19372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:53:05.955247 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 19372
Aug 10 03:53:05 dist-test-slave-pww0 krb5kdc[17006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333985, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:53:05.969574 19127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:05.957148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:49057 (local address 127.27.153.126:39401)
0810 03:53:05.957297 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:05.957302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:05.957321 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:53:05.957848 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:05.957852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:05.957906 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:05.957982 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:05.959337 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:05.959953 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:05.961664 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:05.961896 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:05.962018 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:05.964930 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:05.964935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:05.964938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:05.964969 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:05.966521 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:05.967129 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:05.967133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:05.967134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:05.967232 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:05.967710 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:05.967713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:05.967715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:05.967941 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:05.968005 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:05.969241 (+ 1236us) server_negotiation.cc:299] Negotiation successful
0810 03:53:05.969386 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 03:53:05.971488 19611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:05.955951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39401 (local address 127.27.153.67:49057)
0810 03:53:05.956597 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:05.956654 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:05.957639 (+ 985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:05.958365 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:05.958375 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:05.958676 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:05.959183 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:05.959198 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:05.960100 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:05.960104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:05.961510 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:05.961524 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:05.962223 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:05.962228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:05.962397 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:05.962977 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:05.962995 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:05.964699 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:05.966654 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:05.966662 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:05.966682 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:05.966989 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:05.967331 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:05.967334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:05.967336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:05.967585 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:05.968486 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:05.968493 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:05.968910 (+ 417us) client_negotiation.cc:769] Sending connection context
0810 03:53:05.969962 (+ 1052us) client_negotiation.cc:240] Negotiation successful
0810 03:53:05.970386 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":155,"threads_started":1}
I20260810 03:53:05.973273 19606 heartbeater.cc:347] Connected to a master server at 127.27.153.126:39401
I20260810 03:53:05.973542 19606 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:05.974162 19606 heartbeater.cc:511] Master 127.27.153.126:39401 requested a full tablet report, sending...
I20260810 03:53:05.975909 18772 ts_manager.cc:194] Registered new tserver with Master: 4a040d1ed5ee47deb9d2fba42d885bd6 (127.27.153.67:36685)
I20260810 03:53:05.977097 18772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:49057
I20260810 03:53:05.980748 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:05.957250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41658 (local address 127.27.153.126:39401)
0810 03:53:05.957540 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:05.957549 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:05.958228 (+ 679us) server_negotiation.cc:407] Connection header received
0810 03:53:05.958349 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:05.958352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:05.959762 (+ 1410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:05.959871 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:05.961684 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:05.963446 (+ 1762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:05.967531 (+ 4085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:05.967876 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:05.967973 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:05.969746 (+ 1773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:05.969751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:05.969754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:05.969784 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:05.971791 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:05.973569 (+ 1778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:05.973573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:05.973575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:05.973682 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:05.979546 (+ 5864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:05.979551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:05.979553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:05.979855 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:05.980107 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:05.980386 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 03:53:05.980535 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":116,"threads_started":1}
Aug 10 03:53:06 dist-test-slave-pww0 krb5kdc[17006](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786333986, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:06.372541 19121 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:06.372877 19121 heartbeater.cc:511] Master 127.27.153.126:39401 requested a full tablet report, sending...
I20260810 03:53:06.373903 18772 ts_manager.cc:194] Registered new tserver with Master: 37d8b96b9dc944f999a231f961b5fa76 (127.27.153.65:44597)
I20260810 03:53:06.374706 18772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:58103
I20260810 03:53:06.710528 19362 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:06.710891 19362 heartbeater.cc:511] Master 127.27.153.126:39401 requested a full tablet report, sending...
I20260810 03:53:06.711975 18772 ts_manager.cc:194] Registered new tserver with Master: 11a8abf9e1554d5ba8b4cf6c6463b3b0 (127.27.153.66:35343)
I20260810 03:53:06.712816 18772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:50681
I20260810 03:53:06.720752 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:53:06.734611 19641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:06.722677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41660 (local address 127.27.153.126:39401)
0810 03:53:06.724136 (+ 1459us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:06.724142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:06.724159 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:53:06.724209 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:06.724213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:06.724268 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:06.724364 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:06.725448 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.726398 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.727744 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.728171 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.728275 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:06.730281 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:06.730285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:06.730288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:06.730337 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:06.732255 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:06.732804 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:06.732807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:06.732809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:06.732906 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:06.733185 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:06.733188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:06.733190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:06.733419 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:06.733546 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:06.734282 (+ 736us) server_negotiation.cc:299] Negotiation successful
0810 03:53:06.734426 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1364,"thread_start_us":120,"threads_started":1}
I20260810 03:53:06.739589 18772 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:53:06.744608 18772 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:53:06.774806 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:06.762751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:44597 (local address 127.0.0.1:44312)
0810 03:53:06.765255 (+ 2504us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:06.765268 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:06.765346 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:06.767415 (+ 2069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:06.767423 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:06.767450 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:53:06.767655 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:06.767665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.769668 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.769672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:06.772748 (+ 3076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:06.772762 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.774158 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.774162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:06.774298 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:06.774349 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 03:53:06.774465 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 03:53:06.774620 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2444,"thread_start_us":85,"threads_started":1}
I20260810 03:53:06.776075 19650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:06.762585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:35343 (local address 127.0.0.1:39820)
0810 03:53:06.764983 (+ 2398us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:06.764996 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:06.765078 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:06.767793 (+ 2715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:06.767796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:06.767816 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:53:06.767912 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:06.767919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.772291 (+ 4372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.772295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:06.773521 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:06.773533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.775081 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.775085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:06.775182 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:06.775217 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 03:53:06.775284 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 03:53:06.775354 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2332,"thread_start_us":121,"threads_started":1}
I20260810 03:53:06.776633 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:06.763148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44312 (local address 127.27.153.65:44597)
0810 03:53:06.766718 (+ 3570us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:06.766726 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:06.766752 (+ 26us) server_negotiation.cc:407] Connection header received
0810 03:53:06.766833 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:06.766839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:06.767052 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:06.767207 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:53:06.768294 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.769499 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.772912 (+ 3413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.773810 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.773919 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:06.774082 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:06.776329 (+ 2247us) server_negotiation.cc:299] Negotiation successful
0810 03:53:06.776455 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3434,"thread_start_us":109,"threads_started":1}
I20260810 03:53:06.776824 19656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:06.763050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39820 (local address 127.27.153.66:35343)
0810 03:53:06.765602 (+ 2552us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:06.765610 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:06.765636 (+ 26us) server_negotiation.cc:407] Connection header received
0810 03:53:06.765737 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:06.765743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:06.765994 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:06.766155 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:53:06.771018 (+ 4863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.772154 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.773903 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.774767 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.774874 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:06.775015 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:06.776565 (+ 1550us) server_negotiation.cc:299] Negotiation successful
0810 03:53:06.776682 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2429,"thread_start_us":118,"threads_started":1}
I20260810 03:53:06.780536 19243 tablet_service.cc:1467] Processing CreateTablet for tablet f39c3a0a369c471eaafdf76f2bae6314 (DEFAULT_TABLE table=test-table [id=6902123f603847888dbec2cc9b074ac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:53:06.781509 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:06.761332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:36685 (local address 127.0.0.1:38908)
0810 03:53:06.764477 (+ 3145us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:06.764546 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:06.764712 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:06.776939 (+ 12227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:06.776942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:06.776969 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:53:06.777071 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:06.777077 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.778769 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.778773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:06.779921 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:06.779936 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.781101 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.781105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:06.781200 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:06.781230 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 03:53:06.781283 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 03:53:06.781365 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2919,"thread_start_us":111,"threads_started":1}
I20260810 03:53:06.782181 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:06.761493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38908 (local address 127.27.153.67:36685)
0810 03:53:06.775403 (+ 13910us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:06.775411 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:06.775440 (+ 29us) server_negotiation.cc:407] Connection header received
0810 03:53:06.775550 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:06.775557 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:06.775843 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:06.777257 (+ 1414us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:53:06.777415 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.778476 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.780090 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.780953 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.781722 (+ 769us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:06.781886 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:06.781942 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 03:53:06.782044 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13773,"thread_start_us":99,"threads_started":1}
I20260810 03:53:06.784281 19004 tablet_service.cc:1467] Processing CreateTablet for tablet f39c3a0a369c471eaafdf76f2bae6314 (DEFAULT_TABLE table=test-table [id=6902123f603847888dbec2cc9b074ac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:53:06.785590 19004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f39c3a0a369c471eaafdf76f2bae6314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:53:06.785682 19495 tablet_service.cc:1467] Processing CreateTablet for tablet f39c3a0a369c471eaafdf76f2bae6314 (DEFAULT_TABLE table=test-table [id=6902123f603847888dbec2cc9b074ac0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:53:06.786976 19495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f39c3a0a369c471eaafdf76f2bae6314. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 03:53:06 dist-test-slave-pww0 krb5kdc[17006](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333986, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for HTTP/127.27.153.84@KRBTEST.COM
I20260810 03:53:06.790906 19243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f39c3a0a369c471eaafdf76f2bae6314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:53:06.803937 19662 tablet_bootstrap.cc:492] T f39c3a0a369c471eaafdf76f2bae6314 P 4a040d1ed5ee47deb9d2fba42d885bd6: Bootstrap starting.
I20260810 03:53:06.806804 19662 tablet_bootstrap.cc:654] T f39c3a0a369c471eaafdf76f2bae6314 P 4a040d1ed5ee47deb9d2fba42d885bd6: Neither blocks nor log segments found. Creating new log.
I20260810 03:53:06.807974 19662 log.cc:824] T f39c3a0a369c471eaafdf76f2bae6314 P 4a040d1ed5ee47deb9d2fba42d885bd6: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:06.812191 19662 tablet_bootstrap.cc:492] T f39c3a0a369c471eaafdf76f2bae6314 P 4a040d1ed5ee47deb9d2fba42d885bd6: No bootstrap required, opened a new log
I20260810 03:53:06.812505 19662 ts_tablet_manager.cc:1397] T f39c3a0a369c471eaafdf76f2bae6314 P 4a040d1ed5ee47deb9d2fba42d885bd6: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 03:53:06.817529 19662 raft_consensus.cc:348] T f39c3a0a369c471eaafdf76f2bae6314 P 4a040d1ed5ee47deb9d2fba42d885bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a040d1ed5ee47deb9d2fba42d885bd6" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 36685 } } peers { permanent_uuid: "11a8abf9e1554d5ba8b4cf6c6463b3b0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 35343 } } peers { permanent_uuid: "37d8b96b9dc944f999a231f961b5fa76" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 44597 } }
I20260810 03:53:06.817902 19662 raft_consensus.cc:374] T f39c3a0a369c471eaafdf76f2bae6314 P 4a040d1ed5ee47deb9d2fba42d885bd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:53:06.818003 19662 raft_consensus.cc:729] T f39c3a0a369c471eaafdf76f2bae6314 P 4a040d1ed5ee47deb9d2fba42d885bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a040d1ed5ee47deb9d2fba42d885bd6, State: Initialized, Role: FOLLOWER
I20260810 03:53:06.818614 19662 consensus_queue.cc:260] T f39c3a0a369c471eaafdf76f2bae6314 P 4a040d1ed5ee47deb9d2fba42d885bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a040d1ed5ee47deb9d2fba42d885bd6" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 36685 } } peers { permanent_uuid: "11a8abf9e1554d5ba8b4cf6c6463b3b0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 35343 } } peers { permanent_uuid: "37d8b96b9dc944f999a231f961b5fa76" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 44597 } }
I20260810 03:53:06.820112 19662 ts_tablet_manager.cc:1428] T f39c3a0a369c471eaafdf76f2bae6314 P 4a040d1ed5ee47deb9d2fba42d885bd6: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260810 03:53:06.820603 19606 heartbeater.cc:503] Master 127.27.153.126:39401 was elected leader, sending a full tablet report...
I20260810 03:53:06.825054 19665 tablet_bootstrap.cc:492] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0: Bootstrap starting.
I20260810 03:53:06.825977 19660 tablet_bootstrap.cc:492] T f39c3a0a369c471eaafdf76f2bae6314 P 37d8b96b9dc944f999a231f961b5fa76: Bootstrap starting.
I20260810 03:53:06.829430 19665 tablet_bootstrap.cc:654] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0: Neither blocks nor log segments found. Creating new log.
I20260810 03:53:06.829430 19660 tablet_bootstrap.cc:654] T f39c3a0a369c471eaafdf76f2bae6314 P 37d8b96b9dc944f999a231f961b5fa76: Neither blocks nor log segments found. Creating new log.
I20260810 03:53:06.830504 19665 log.cc:824] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:06.830504 19660 log.cc:824] T f39c3a0a369c471eaafdf76f2bae6314 P 37d8b96b9dc944f999a231f961b5fa76: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:06.834020 19665 tablet_bootstrap.cc:492] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0: No bootstrap required, opened a new log
I20260810 03:53:06.836539 19665 ts_tablet_manager.cc:1397] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260810 03:53:06.837180 19660 tablet_bootstrap.cc:492] T f39c3a0a369c471eaafdf76f2bae6314 P 37d8b96b9dc944f999a231f961b5fa76: No bootstrap required, opened a new log
I20260810 03:53:06.837428 19660 ts_tablet_manager.cc:1397] T f39c3a0a369c471eaafdf76f2bae6314 P 37d8b96b9dc944f999a231f961b5fa76: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260810 03:53:06.842187 19665 raft_consensus.cc:348] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a040d1ed5ee47deb9d2fba42d885bd6" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 36685 } } peers { permanent_uuid: "11a8abf9e1554d5ba8b4cf6c6463b3b0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 35343 } } peers { permanent_uuid: "37d8b96b9dc944f999a231f961b5fa76" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 44597 } }
I20260810 03:53:06.842631 19665 raft_consensus.cc:374] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:53:06.842742 19665 raft_consensus.cc:729] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11a8abf9e1554d5ba8b4cf6c6463b3b0, State: Initialized, Role: FOLLOWER
I20260810 03:53:06.842773 19660 raft_consensus.cc:348] T f39c3a0a369c471eaafdf76f2bae6314 P 37d8b96b9dc944f999a231f961b5fa76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a040d1ed5ee47deb9d2fba42d885bd6" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 36685 } } peers { permanent_uuid: "11a8abf9e1554d5ba8b4cf6c6463b3b0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 35343 } } peers { permanent_uuid: "37d8b96b9dc944f999a231f961b5fa76" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 44597 } }
I20260810 03:53:06.843079 19660 raft_consensus.cc:374] T f39c3a0a369c471eaafdf76f2bae6314 P 37d8b96b9dc944f999a231f961b5fa76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:53:06.843159 19660 raft_consensus.cc:729] T f39c3a0a369c471eaafdf76f2bae6314 P 37d8b96b9dc944f999a231f961b5fa76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37d8b96b9dc944f999a231f961b5fa76, State: Initialized, Role: FOLLOWER
I20260810 03:53:06.843530 19665 consensus_queue.cc:260] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a040d1ed5ee47deb9d2fba42d885bd6" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 36685 } } peers { permanent_uuid: "11a8abf9e1554d5ba8b4cf6c6463b3b0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 35343 } } peers { permanent_uuid: "37d8b96b9dc944f999a231f961b5fa76" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 44597 } }
I20260810 03:53:06.843652 19660 consensus_queue.cc:260] T f39c3a0a369c471eaafdf76f2bae6314 P 37d8b96b9dc944f999a231f961b5fa76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a040d1ed5ee47deb9d2fba42d885bd6" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 36685 } } peers { permanent_uuid: "11a8abf9e1554d5ba8b4cf6c6463b3b0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 35343 } } peers { permanent_uuid: "37d8b96b9dc944f999a231f961b5fa76" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 44597 } }
I20260810 03:53:06.844849 19665 ts_tablet_manager.cc:1428] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260810 03:53:06.844851 19660 ts_tablet_manager.cc:1428] T f39c3a0a369c471eaafdf76f2bae6314 P 37d8b96b9dc944f999a231f961b5fa76: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 03:53:06.845286 19362 heartbeater.cc:503] Master 127.27.153.126:39401 was elected leader, sending a full tablet report...
I20260810 03:53:06.845494 19121 heartbeater.cc:503] Master 127.27.153.126:39401 was elected leader, sending a full tablet report...
W20260810 03:53:06.917601 19366 tablet.cc:2369] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:53:06.930724 19670 raft_consensus.cc:482] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:53:06.931010 19670 raft_consensus.cc:504] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a040d1ed5ee47deb9d2fba42d885bd6" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 36685 } } peers { permanent_uuid: "11a8abf9e1554d5ba8b4cf6c6463b3b0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 35343 } } peers { permanent_uuid: "37d8b96b9dc944f999a231f961b5fa76" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 44597 } }
I20260810 03:53:06.932600 19670 leader_election.cc:288] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a040d1ed5ee47deb9d2fba42d885bd6 (127.27.153.67:36685), 37d8b96b9dc944f999a231f961b5fa76 (127.27.153.65:44597)
I20260810 03:53:06.936977 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:06.933044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:36439 (local address 127.27.153.67:36685)
0810 03:53:06.933273 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:06.933278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:06.933365 (+ 87us) server_negotiation.cc:407] Connection header received
0810 03:53:06.933559 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:06.933562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:06.933612 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:06.933656 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:53:06.934049 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.934795 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.935963 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.936403 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.936526 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:06.936583 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:06.936759 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 03:53:06.936837 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260810 03:53:06.937004 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:06.932822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:36685 (local address 127.27.153.66:36439)
0810 03:53:06.933273 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:06.933288 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:06.933442 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:06.933807 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:06.933808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:06.933823 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:53:06.933925 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:06.933931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.934931 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.934935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:06.935840 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:06.935849 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.936527 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.936531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:06.936630 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:06.936658 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 03:53:06.936738 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 03:53:06.936814 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":141,"thread_start_us":84,"threads_started":1}
I20260810 03:53:06.937582 19659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:06.933037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:53291 (local address 127.27.153.65:44597)
0810 03:53:06.933176 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:06.933181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:06.933368 (+ 187us) server_negotiation.cc:407] Connection header received
0810 03:53:06.933517 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:06.933521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:06.933573 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:06.933647 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:53:06.933960 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.934770 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.936364 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.936889 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.937001 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:06.937051 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:06.937375 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 03:53:06.937463 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:53:06.937878 19516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f39c3a0a369c471eaafdf76f2bae6314" candidate_uuid: "11a8abf9e1554d5ba8b4cf6c6463b3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a040d1ed5ee47deb9d2fba42d885bd6" is_pre_election: true
I20260810 03:53:06.938122 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:06.932823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:44597 (local address 127.27.153.66:53291)
0810 03:53:06.933248 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:06.933264 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:06.933404 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:06.933667 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:06.933670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:06.933687 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:53:06.933847 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:06.933853 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.934894 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.934898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:06.936223 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:06.936234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.937116 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.937119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:06.937240 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:06.937263 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 03:53:06.937903 (+ 640us) client_negotiation.cc:240] Negotiation successful
0810 03:53:06.937984 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":3,"thread_start_us":161,"threads_started":1}
I20260810 03:53:06.938244 19516 raft_consensus.cc:2454] T f39c3a0a369c471eaafdf76f2bae6314 P 4a040d1ed5ee47deb9d2fba42d885bd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11a8abf9e1554d5ba8b4cf6c6463b3b0 in term 0.
I20260810 03:53:06.938716 19036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f39c3a0a369c471eaafdf76f2bae6314" candidate_uuid: "11a8abf9e1554d5ba8b4cf6c6463b3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37d8b96b9dc944f999a231f961b5fa76" is_pre_election: true
I20260810 03:53:06.938987 19036 raft_consensus.cc:2454] T f39c3a0a369c471eaafdf76f2bae6314 P 37d8b96b9dc944f999a231f961b5fa76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 11a8abf9e1554d5ba8b4cf6c6463b3b0 in term 0.
I20260810 03:53:06.939507 19155 leader_election.cc:302] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11a8abf9e1554d5ba8b4cf6c6463b3b0, 4a040d1ed5ee47deb9d2fba42d885bd6; no voters:
I20260810 03:53:06.940026 19670 raft_consensus.cc:2781] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:53:06.940107 19670 raft_consensus.cc:482] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:53:06.940146 19670 raft_consensus.cc:3037] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:53:06.941515 19670 raft_consensus.cc:504] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a040d1ed5ee47deb9d2fba42d885bd6" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 36685 } } peers { permanent_uuid: "11a8abf9e1554d5ba8b4cf6c6463b3b0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 35343 } } peers { permanent_uuid: "37d8b96b9dc944f999a231f961b5fa76" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 44597 } }
I20260810 03:53:06.941926 19670 leader_election.cc:288] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0 [CANDIDATE]: Term 1 election: Requested vote from peers 4a040d1ed5ee47deb9d2fba42d885bd6 (127.27.153.67:36685), 37d8b96b9dc944f999a231f961b5fa76 (127.27.153.65:44597)
I20260810 03:53:06.942476 19036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f39c3a0a369c471eaafdf76f2bae6314" candidate_uuid: "11a8abf9e1554d5ba8b4cf6c6463b3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37d8b96b9dc944f999a231f961b5fa76"
I20260810 03:53:06.942533 19516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f39c3a0a369c471eaafdf76f2bae6314" candidate_uuid: "11a8abf9e1554d5ba8b4cf6c6463b3b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a040d1ed5ee47deb9d2fba42d885bd6"
I20260810 03:53:06.942605 19036 raft_consensus.cc:3037] T f39c3a0a369c471eaafdf76f2bae6314 P 37d8b96b9dc944f999a231f961b5fa76 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:53:06.942657 19516 raft_consensus.cc:3037] T f39c3a0a369c471eaafdf76f2bae6314 P 4a040d1ed5ee47deb9d2fba42d885bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:53:06.943945 19036 raft_consensus.cc:2454] T f39c3a0a369c471eaafdf76f2bae6314 P 37d8b96b9dc944f999a231f961b5fa76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11a8abf9e1554d5ba8b4cf6c6463b3b0 in term 1.
I20260810 03:53:06.944039 19516 raft_consensus.cc:2454] T f39c3a0a369c471eaafdf76f2bae6314 P 4a040d1ed5ee47deb9d2fba42d885bd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 11a8abf9e1554d5ba8b4cf6c6463b3b0 in term 1.
I20260810 03:53:06.944594 19158 leader_election.cc:302] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11a8abf9e1554d5ba8b4cf6c6463b3b0, 37d8b96b9dc944f999a231f961b5fa76; no voters:
I20260810 03:53:06.944912 19670 raft_consensus.cc:2781] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:53:06.945178 19670 raft_consensus.cc:686] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0 [term 1 LEADER]: Becoming Leader. State: Replica: 11a8abf9e1554d5ba8b4cf6c6463b3b0, State: Running, Role: LEADER
I20260810 03:53:06.945580 19670 consensus_queue.cc:237] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a040d1ed5ee47deb9d2fba42d885bd6" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 36685 } } peers { permanent_uuid: "11a8abf9e1554d5ba8b4cf6c6463b3b0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 35343 } } peers { permanent_uuid: "37d8b96b9dc944f999a231f961b5fa76" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 44597 } }
I20260810 03:53:06.949359 18771 catalog_manager.cc:5495] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 11a8abf9e1554d5ba8b4cf6c6463b3b0 (127.27.153.66). New cstate: current_term: 1 leader_uuid: "11a8abf9e1554d5ba8b4cf6c6463b3b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a040d1ed5ee47deb9d2fba42d885bd6" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 36685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11a8abf9e1554d5ba8b4cf6c6463b3b0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 35343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37d8b96b9dc944f999a231f961b5fa76" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 44597 } health_report { overall_health: UNKNOWN } } }
W20260810 03:53:06.955183 19608 tablet.cc:2369] T f39c3a0a369c471eaafdf76f2bae6314 P 4a040d1ed5ee47deb9d2fba42d885bd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:53:06.976537 19656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:06.971794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39832 (local address 127.27.153.66:35343)
0810 03:53:06.971918 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:06.971921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:06.971935 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:53:06.972131 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:06.972134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:06.972217 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:06.972348 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:53:06.972610 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.973403 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.974613 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:06.974972 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:06.975103 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:06.975424 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:53:06.976070 (+ 646us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:53:06.976202 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:06.976285 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 03:53:06.976358 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:53:06.984201 19036 raft_consensus.cc:1261] T f39c3a0a369c471eaafdf76f2bae6314 P 37d8b96b9dc944f999a231f961b5fa76 [term 1 FOLLOWER]: Refusing update from remote peer 11a8abf9e1554d5ba8b4cf6c6463b3b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:53:06.984246 19516 raft_consensus.cc:1261] T f39c3a0a369c471eaafdf76f2bae6314 P 4a040d1ed5ee47deb9d2fba42d885bd6 [term 1 FOLLOWER]: Refusing update from remote peer 11a8abf9e1554d5ba8b4cf6c6463b3b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:53:06.985340 19677 consensus_queue.cc:1035] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a040d1ed5ee47deb9d2fba42d885bd6" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:53:06.985680 19670 consensus_queue.cc:1035] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37d8b96b9dc944f999a231f961b5fa76" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 44597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:53:06.998435 19683 mvcc.cc:205] Tried to move back new op lower bound from 7316824010678616064 to 7316824010532474880. Current Snapshot: MvccSnapshot[applied={T|T < 7316824010678616064}]
I20260810 03:53:06.999081 19687 mvcc.cc:205] Tried to move back new op lower bound from 7316824010678616064 to 7316824010532474880. Current Snapshot: MvccSnapshot[applied={T|T < 7316824010678616064}]
I20260810 03:53:07.004366 19685 mvcc.cc:205] Tried to move back new op lower bound from 7316824010678616064 to 7316824010532474880. Current Snapshot: MvccSnapshot[applied={T|T < 7316824010678616064}]
I20260810 03:53:07.013610 18771 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:53:07.013901 18771 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:53:07.018117 18771 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 055d6244c60e417d9766c9889ca6c090: Sending DeleteTablet for 3 replicas of tablet f39c3a0a369c471eaafdf76f2bae6314
I20260810 03:53:07.019667 19495 tablet_service.cc:1514] Processing DeleteTablet for tablet f39c3a0a369c471eaafdf76f2bae6314 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:53:07 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:38908
I20260810 03:53:07.019878 19243 tablet_service.cc:1514] Processing DeleteTablet for tablet f39c3a0a369c471eaafdf76f2bae6314 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:53:07 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:39820
I20260810 03:53:07.020907 19004 tablet_service.cc:1514] Processing DeleteTablet for tablet f39c3a0a369c471eaafdf76f2bae6314 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:53:07 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:44312
I20260810 03:53:07.022998 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 18841
I20260810 03:53:07.031514 19698 tablet_replica.cc:330] T f39c3a0a369c471eaafdf76f2bae6314 P 4a040d1ed5ee47deb9d2fba42d885bd6: stopping tablet replica
I20260810 03:53:07.031972 19698 raft_consensus.cc:2229] T f39c3a0a369c471eaafdf76f2bae6314 P 4a040d1ed5ee47deb9d2fba42d885bd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:53:07.032424 19698 raft_consensus.cc:2258] T f39c3a0a369c471eaafdf76f2bae6314 P 4a040d1ed5ee47deb9d2fba42d885bd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:53:07.036816 19698 ts_tablet_manager.cc:1905] T f39c3a0a369c471eaafdf76f2bae6314 P 4a040d1ed5ee47deb9d2fba42d885bd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:53:07.045239 19698 ts_tablet_manager.cc:1918] T f39c3a0a369c471eaafdf76f2bae6314 P 4a040d1ed5ee47deb9d2fba42d885bd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:53:07.045534 19698 log.cc:1196] T f39c3a0a369c471eaafdf76f2bae6314 P 4a040d1ed5ee47deb9d2fba42d885bd6: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/f39c3a0a369c471eaafdf76f2bae6314
I20260810 03:53:07.045828 19697 tablet_replica.cc:330] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0: stopping tablet replica
I20260810 03:53:07.046069 19698 ts_tablet_manager.cc:1939] T f39c3a0a369c471eaafdf76f2bae6314 P 4a040d1ed5ee47deb9d2fba42d885bd6: Deleting consensus metadata
I20260810 03:53:07.046324 19697 raft_consensus.cc:2229] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:53:07.046836 19697 raft_consensus.cc:2258] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:53:07.047874 18738 catalog_manager.cc:4841] TS 4a040d1ed5ee47deb9d2fba42d885bd6 (127.27.153.67:36685): tablet f39c3a0a369c471eaafdf76f2bae6314 (table test-table [id=6902123f603847888dbec2cc9b074ac0]) successfully deleted
I20260810 03:53:07.048727 19697 ts_tablet_manager.cc:1905] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:53:07.051973 19697 ts_tablet_manager.cc:1918] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260810 03:53:07.052603 18741 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.153.65:44597 (error 108)
I20260810 03:53:07.053561 19697 log.cc:1196] T f39c3a0a369c471eaafdf76f2bae6314 P 11a8abf9e1554d5ba8b4cf6c6463b3b0: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/f39c3a0a369c471eaafdf76f2bae6314
I20260810 03:53:07.053870 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 19126
I20260810 03:53:07.081007 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:07.080330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:44597 (local address 127.0.0.1:44320)
0810 03:53:07.080773 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:07.080887 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:44597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291}
W20260810 03:53:07.081462 18741 catalog_manager.cc:4584] TS 37d8b96b9dc944f999a231f961b5fa76 (127.27.153.65:44597): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f39c3a0a369c471eaafdf76f2bae6314: Network error: Client connection negotiation failed: client connection to 127.27.153.65:44597: connect: Connection refused (error 111)
I20260810 03:53:07.088553 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 19372
W20260810 03:53:07.088744 19655 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:53:07.087948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:35343 (local address 127.0.0.1:39840)
0810 03:53:07.088556 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:07.088626 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.66:35343: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":506}
I20260810 03:53:07.120501 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 18707
I20260810 03:53:07.139318 28261 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 03:53:07.485121 28261 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 03:53:07.485242 28261 mini_ranger.cc:66] Stopping Ranger...
I20260810 03:53:07.831089 28261 mini_ranger.cc:68] Stopped Ranger
2026-08-10 03:53:07.831 UTC [17071] LOG: received smart shutdown request
2026-08-10 03:53:07.836 UTC [17071] LOG: background worker "logical replication launcher" (PID 17084) exited with exit code 1
2026-08-10 03:53:07.836 UTC [17076] LOG: shutting down
2026-08-10 03:53:08.013 UTC [17071] LOG: database system is shut down
2026-08-10T03:53:08Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52011 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:53:08 dist-test-slave-pww0 krb5kdc[19725](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:53:08 dist-test-slave-pww0 krb5kdc[19725](info): set up 2 sockets
Aug 10 03:53:08 dist-test-slave-pww0 krb5kdc[19725](info): commencing operation
krb5kdc: starting...
W20260810 03:53:09.343501 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.191s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:53:09 dist-test-slave-pww0 krb5kdc[19725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:53:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:53:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:53:09.477016 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:33531
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:46333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:33531,127.27.153.125:36189,127.27.153.124:35765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:53:09.594141 19748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:09.594429 19748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:09.594472 19748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:09.598598 19748 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:53:09.598676 19748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:09.598696 19748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:09.598712 19748 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:53:09.598726 19748 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:53:09.604387 19748 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:46333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:33531,127.27.153.125:36189,127.27.153.124:35765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:33531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:09.605520 19748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:09.606623 19748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:09.615509 19765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:09.615502 19758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:09.615502 19759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:09.615964 19748 server_base.cc:1034] running on GCE node
I20260810 03:53:09.616618 19748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:09.617797 19748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:09.618959 19748 hybrid_clock.cc:648] HybridClock initialized: now 1786333989618944 us; error 32 us; skew 500 ppm
Aug 10 03:53:09 dist-test-slave-pww0 krb5kdc[19725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333989, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:09.621675 19748 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:53:09.622586 19748 webserver.cc:460] Webserver started at http://127.27.153.126:38997/ using document root <none> and password file <none>
I20260810 03:53:09.623114 19748 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:09.623174 19748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:09.623355 19748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:53:09.625540 19748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "bbc7378be5db47d3b7959f8d29d9c668"
format_stamp: "Formatted at 2026-08-10 03:53:09 on dist-test-slave-pww0"
server_key: "a8da8fc3ea069e5d1f3ff568ca9852b3"
server_key_iv: "3e2c1bf879bec813c76a4112a8111641"
server_key_version: "encryptionkey@0"
I20260810 03:53:09.626087 19748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "bbc7378be5db47d3b7959f8d29d9c668"
format_stamp: "Formatted at 2026-08-10 03:53:09 on dist-test-slave-pww0"
server_key: "a8da8fc3ea069e5d1f3ff568ca9852b3"
server_key_iv: "3e2c1bf879bec813c76a4112a8111641"
server_key_version: "encryptionkey@0"
I20260810 03:53:09.630168 19748 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260810 03:53:09.632879 19775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:09.634296 19748 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:53:09.634502 19748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "bbc7378be5db47d3b7959f8d29d9c668"
format_stamp: "Formatted at 2026-08-10 03:53:09 on dist-test-slave-pww0"
server_key: "a8da8fc3ea069e5d1f3ff568ca9852b3"
server_key_iv: "3e2c1bf879bec813c76a4112a8111641"
server_key_version: "encryptionkey@0"
I20260810 03:53:09.634624 19748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:09.664960 19748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:09.668365 19748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:09.668562 19748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:09.675889 19748 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:33531
I20260810 03:53:09.675925 19863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:33531 every 8 connection(s)
I20260810 03:53:09.676913 19748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:53:09.679920 19864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:53:09.684026 19864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33531 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:53:09.685361 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 19748
I20260810 03:53:09.685482 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:53:09.685760 28261 external_mini_cluster.cc:1428] Setting key 82f0a5e9c02cb4773515df42e0b27899
Aug 10 03:53:09 dist-test-slave-pww0 krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333989, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.153.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.125@KRBTEST.COM" created.
I20260810 03:53:09.699904 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:09.686534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:33531 (local address 127.0.0.1:53444)
0810 03:53:09.687236 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:09.687291 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:09.688229 (+ 938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:09.689075 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:09.689080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:09.689432 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:09.689699 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:09.689713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:09.690908 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:09.690911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:09.691697 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:09.691707 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:09.692335 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:09.692340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:09.692504 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:09.693127 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:09.693145 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:09.694736 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:09.696648 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:09.696668 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:09.696677 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:09.696939 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:09.697388 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:09.697391 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:09.697393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:09.697562 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:09.698204 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:09.698212 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:09.698409 (+ 197us) client_negotiation.cc:769] Sending connection context
0810 03:53:09.698601 (+ 192us) client_negotiation.cc:240] Negotiation successful
0810 03:53:09.699024 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":85,"spinlock_wait_cycles":9984,"thread_start_us":169,"threads_started":1}
I20260810 03:53:09.699911 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:09.686534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53444 (local address 127.27.153.126:33531)
0810 03:53:09.687114 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:09.687134 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:09.687414 (+ 280us) server_negotiation.cc:407] Connection header received
0810 03:53:09.688579 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:09.688585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:09.688853 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:09.689065 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:09.689864 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:09.690775 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:09.691812 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:09.692181 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:09.692341 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:09.694906 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:09.694913 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:09.694924 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:09.694948 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:09.696505 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:09.697119 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:09.697126 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:09.697128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:09.697255 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:09.697656 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:09.697659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:09.697660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:09.698075 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:09.698188 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:09.698669 (+ 481us) server_negotiation.cc:299] Negotiation successful
0810 03:53:09.698926 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":387,"spinlock_wait_cycles":61184,"thread_start_us":169,"threads_started":1}
I20260810 03:53:09.702436 19864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:53:09.703233 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:09.702934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:36189 (local address 127.0.0.1:40088)
0810 03:53:09.703060 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:09.703147 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.125:36189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 03:53:09.703545 19776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.153.125:36189: connect: Connection refused (error 111)
I20260810 03:53:09.704432 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:09.704240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:36189 (local address 127.0.0.1:40104)
0810 03:53:09.704334 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:09.704378 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.125:36189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 03:53:09.704569 19864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.125:36189: Network error: Client connection negotiation failed: client connection to 127.27.153.125:36189: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.125.keytab.
Entry for principal kudu/127.27.153.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.125.keytab.
I20260810 03:53:09.714833 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.125:36189
--webserver_interface=127.27.153.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.125.keytab
--principal=kudu/127.27.153.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:46333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:33531,127.27.153.125:36189,127.27.153.124:35765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:53:09.753865 19864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } attempt: 1
I20260810 03:53:09.754877 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:09.754508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:36189 (local address 127.0.0.1:40106)
0810 03:53:09.754714 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:09.754789 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.125:36189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260810 03:53:09.755678 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:09.755387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:36189 (local address 127.0.0.1:40118)
0810 03:53:09.755544 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:09.755603 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.125:36189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 03:53:09.755801 19864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.125:36189: Network error: Client connection negotiation failed: client connection to 127.27.153.125:36189: connect: Connection refused (error 111)
I20260810 03:53:09.823967 19864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } attempt: 2
I20260810 03:53:09.824894 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:09.824565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:36189 (local address 127.0.0.1:40128)
0810 03:53:09.824744 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:09.824813 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.125:36189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 03:53:09.825623 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:09.825402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:36189 (local address 127.0.0.1:40138)
0810 03:53:09.825507 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:09.825557 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.125:36189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 03:53:09.825743 19864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.125:36189: Network error: Client connection negotiation failed: client connection to 127.27.153.125:36189: connect: Connection refused (error 111)
W20260810 03:53:09.833264 19881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:09.833596 19881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:09.833654 19881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:09.837834 19881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:53:09.837913 19881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:09.837940 19881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:09.837965 19881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:53:09.837996 19881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:53:09.843739 19881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:46333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:33531,127.27.153.125:36189,127.27.153.124:35765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.125.keytab
--principal=kudu/127.27.153.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.125:36189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.153.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:09.844909 19881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:09.845988 19881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:09.855006 19892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:09.855069 19881 server_base.cc:1034] running on GCE node
W20260810 03:53:09.855006 19891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:09.855006 19894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:09.856096 19881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:09.857317 19881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:09.858482 19881 hybrid_clock.cc:648] HybridClock initialized: now 1786333989858466 us; error 34 us; skew 500 ppm
Aug 10 03:53:09 dist-test-slave-pww0 krb5kdc[19725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333989, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:09.861333 19881 init.cc:377] Logged in from keytab as kudu/127.27.153.125@KRBTEST.COM (short username kudu)
I20260810 03:53:09.862308 19881 webserver.cc:460] Webserver started at http://127.27.153.125:32967/ using document root <none> and password file <none>
I20260810 03:53:09.862886 19881 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:09.862958 19881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:09.863176 19881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:53:09.865386 19881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-1/data/instance:
uuid: "1ee62bde015645f1bd0777a3de36d3cb"
format_stamp: "Formatted at 2026-08-10 03:53:09 on dist-test-slave-pww0"
server_key: "6fa76fded18d1f69b518b3b51108d388"
server_key_iv: "b446b331b6738c5de7e3f3d6122767cc"
server_key_version: "encryptionkey@0"
I20260810 03:53:09.865995 19881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-1/wal/instance:
uuid: "1ee62bde015645f1bd0777a3de36d3cb"
format_stamp: "Formatted at 2026-08-10 03:53:09 on dist-test-slave-pww0"
server_key: "6fa76fded18d1f69b518b3b51108d388"
server_key_iv: "b446b331b6738c5de7e3f3d6122767cc"
server_key_version: "encryptionkey@0"
I20260810 03:53:09.870092 19881 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:53:09.872757 19905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:09.874100 19881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:53:09.874244 19881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-1/wal
uuid: "1ee62bde015645f1bd0777a3de36d3cb"
format_stamp: "Formatted at 2026-08-10 03:53:09 on dist-test-slave-pww0"
server_key: "6fa76fded18d1f69b518b3b51108d388"
server_key_iv: "b446b331b6738c5de7e3f3d6122767cc"
server_key_version: "encryptionkey@0"
I20260810 03:53:09.874357 19881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:09.888257 19881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:09.891232 19881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:09.891424 19881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:09.898449 19881 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.125:36189
I20260810 03:53:09.898725 20001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.125:36189 every 8 connection(s)
I20260810 03:53:09.899495 19881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-1/data/info.pb
I20260810 03:53:09.902411 20002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:53:09.902545 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 19881
I20260810 03:53:09.902673 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-1/wal/instance
I20260810 03:53:09.902937 28261 external_mini_cluster.cc:1428] Setting key 458d45f4fba735439f32999f3b22f9a2
I20260810 03:53:09.907058 20002 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33531 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.153.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.124@KRBTEST.COM" created.
I20260810 03:53:09.916893 19864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } attempt: 3
Aug 10 03:53:09 dist-test-slave-pww0 krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333989, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.125@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:53:09.922451 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:09.909106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53446 (local address 127.27.153.126:33531)
0810 03:53:09.909235 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:09.909239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:09.910399 (+ 1160us) server_negotiation.cc:407] Connection header received
0810 03:53:09.911354 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:09.911357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:09.911422 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:09.911528 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:09.912961 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:09.913543 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:09.914845 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:09.915097 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:09.915196 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:09.917869 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:09.917872 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:09.917875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:09.917902 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:09.920007 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:09.920644 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:09.920648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:09.920650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:09.920748 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:09.921155 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:09.921158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:09.921160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:09.921392 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:09.921482 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:09.922134 (+ 652us) server_negotiation.cc:299] Negotiation successful
0810 03:53:09.922267 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 03:53:09 dist-test-slave-pww0 krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333989, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for kudu/127.27.153.125@KRBTEST.COM
I20260810 03:53:09.923415 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:09.909657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:33531 (local address 127.0.0.1:53446)
0810 03:53:09.910214 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:09.910259 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:09.911135 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:09.911708 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:09.911716 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:09.912106 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:09.912789 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:09.912806 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:09.913684 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:09.913687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:09.914696 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:09.914710 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:09.915256 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:09.915260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:09.915481 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:09.916080 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:09.916097 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:09.917668 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:09.920119 (+ 2451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:09.920126 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:09.920150 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:09.920521 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:09.920854 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:09.920858 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:09.920860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:09.921055 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:09.921527 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:09.921532 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:09.921845 (+ 313us) client_negotiation.cc:769] Sending connection context
0810 03:53:09.922056 (+ 211us) client_negotiation.cc:240] Negotiation successful
0810 03:53:09.922396 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":166,"threads_started":1}
I20260810 03:53:09.925541 20002 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:53:09.927217 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:09.917416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:36189 (local address 127.0.0.1:40144)
0810 03:53:09.917642 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:09.917657 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:09.917736 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:09.918406 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:09.918409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:09.918586 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:09.918688 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:09.918693 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:09.920085 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:09.920088 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:09.920650 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:09.920660 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:09.921378 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:09.921383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:09.921500 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:09.921952 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:09.921966 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:09.923434 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:09.925419 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:09.925422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:09.925424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:09.925650 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:09.925984 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:09.925987 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:09.925989 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:09.926082 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:09.926684 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:09.926687 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:09.926801 (+ 114us) client_negotiation.cc:769] Sending connection context
0810 03:53:09.926906 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 03:53:09.927053 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260810 03:53:09.927316 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:09.917540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40144 (local address 127.27.153.125:36189)
0810 03:53:09.917979 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:09.917992 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:09.918017 (+ 25us) server_negotiation.cc:407] Connection header received
0810 03:53:09.918060 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:09.918066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:09.918277 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:09.918420 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:09.918797 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:09.919946 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:09.920791 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:09.921240 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:09.921356 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:09.923632 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:09.923643 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:09.923658 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:09.923684 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:09.925271 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:09.925749 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:09.925755 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:09.925756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:09.925891 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:09.926265 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:09.926267 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:09.926268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:09.926507 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:09.926662 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:09.927007 (+ 345us) server_negotiation.cc:299] Negotiation successful
0810 03:53:09.927147 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":128,"threads_started":1}
I20260810 03:53:09.929133 19864 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:53:09.929937 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:09.929679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:35765 (local address 127.0.0.1:35310)
0810 03:53:09.929810 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:09.929868 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:35765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 03:53:09.930754 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:09.930399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:35765 (local address 127.0.0.1:35324)
0810 03:53:09.930587 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:09.930681 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:35765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
Aug 10 03:53:09 dist-test-slave-pww0 krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333989, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.125@KRBTEST.COM for kudu/127.27.153.125@KRBTEST.COM
W20260810 03:53:09.930907 19864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.124:35765: Network error: Client connection negotiation failed: client connection to 127.27.153.124:35765: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.124.keytab.
Entry for principal kudu/127.27.153.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.124.keytab.
I20260810 03:53:09.932608 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.124:35765
--webserver_interface=127.27.153.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.124.keytab
--principal=kudu/127.27.153.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:46333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:33531,127.27.153.125:36189,127.27.153.124:35765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:53:09.934852 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:09.926017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:36189 (local address 127.0.0.1:40148)
0810 03:53:09.926121 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:09.926134 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:09.926197 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:09.926798 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:09.926800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:09.926923 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:09.927016 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:09.927020 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:09.928023 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:09.928026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:09.928616 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:09.928626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:09.929084 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:09.929087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:09.929196 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:09.929673 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:09.929687 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:09.931132 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:09.933433 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:09.933437 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:09.933439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:09.933688 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:09.933948 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:09.933951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:09.933953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:09.934042 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:09.934448 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:09.934451 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:09.934535 (+ 84us) client_negotiation.cc:769] Sending connection context
0810 03:53:09.934615 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 03:53:09.934732 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 03:53:09.935014 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:09.926257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40148 (local address 127.27.153.125:36189)
0810 03:53:09.926588 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:09.926598 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:09.926612 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:53:09.926646 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:09.926648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:09.926693 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:09.926780 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:09.927117 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:09.927923 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:09.928722 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:09.928975 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:09.929070 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:09.931281 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:09.931297 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:09.931300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:09.931322 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:09.933313 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:09.933789 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:09.933791 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:09.933792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:09.933870 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:09.934129 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:09.934133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:09.934134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:09.934350 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:09.934450 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:09.934692 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 03:53:09.934810 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":119,"threads_started":1}
I20260810 03:53:09.935786 20002 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:53:09.936492 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:09.936218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:35765 (local address 127.0.0.1:35328)
0810 03:53:09.936336 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:09.936426 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:35765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 03:53:09.936707 19907 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.153.124:35765: connect: Connection refused (error 111)
I20260810 03:53:09.937548 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:09.937322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:35765 (local address 127.0.0.1:35338)
0810 03:53:09.937437 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:09.937489 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:35765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 03:53:09.937763 20002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.124:35765: Network error: Client connection negotiation failed: client connection to 127.27.153.124:35765: connect: Connection refused (error 111)
I20260810 03:53:09.962072 19864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } attempt: 1
I20260810 03:53:09.963038 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:09.962718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:35765 (local address 127.0.0.1:35342)
0810 03:53:09.962887 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:09.962953 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:35765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 03:53:09.963773 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:09.963576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:35765 (local address 127.0.0.1:35356)
0810 03:53:09.963668 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:09.963714 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:35765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 03:53:09.963933 19864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.124:35765: Network error: Client connection negotiation failed: client connection to 127.27.153.124:35765: connect: Connection refused (error 111)
I20260810 03:53:09.987057 20002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } attempt: 1
I20260810 03:53:09.988080 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:09.987730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:35765 (local address 127.0.0.1:35362)
0810 03:53:09.987904 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:09.987981 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:35765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 03:53:09.988837 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:09.988607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:35765 (local address 127.0.0.1:35372)
0810 03:53:09.988716 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:09.988769 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:35765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 03:53:09.989007 20002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.124:35765: Network error: Client connection negotiation failed: client connection to 127.27.153.124:35765: connect: Connection refused (error 111)
I20260810 03:53:10.039124 19864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } attempt: 2
I20260810 03:53:10.040019 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:10.039711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:35765 (local address 127.0.0.1:35374)
0810 03:53:10.039862 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:10.039933 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:35765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 03:53:10.040772 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:10.040570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:35765 (local address 127.0.0.1:35384)
0810 03:53:10.040663 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:10.040712 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:35765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 03:53:10.040934 19864 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.124:35765: Network error: Client connection negotiation failed: client connection to 127.27.153.124:35765: connect: Connection refused (error 111)
W20260810 03:53:10.049633 20015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:10.049949 20015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:10.050010 20015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:10.054466 20015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:53:10.054549 20015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:10.054576 20015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:10.054598 20015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:53:10.054628 20015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:53:10.057178 20002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } attempt: 2
I20260810 03:53:10.058035 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:10.057711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:35765 (local address 127.0.0.1:35400)
0810 03:53:10.057869 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:10.057940 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:35765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 03:53:10.058739 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:10.058525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:35765 (local address 127.0.0.1:35416)
0810 03:53:10.058616 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:10.058676 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:35765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 03:53:10.058857 20002 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.124:35765: Network error: Client connection negotiation failed: client connection to 127.27.153.124:35765: connect: Connection refused (error 111)
I20260810 03:53:10.060765 20015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:46333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:33531,127.27.153.125:36189,127.27.153.124:35765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.124.keytab
--principal=kudu/127.27.153.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.124:35765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.153.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:10.061854 20015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:10.062852 20015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:10.071316 20028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:10.071329 20021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:10.071410 20015 server_base.cc:1034] running on GCE node
W20260810 03:53:10.071316 20022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:10.072141 20015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:10.073259 20015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:10.074432 20015 hybrid_clock.cc:648] HybridClock initialized: now 1786333990074413 us; error 31 us; skew 500 ppm
Aug 10 03:53:10 dist-test-slave-pww0 krb5kdc[19725](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333990, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:10.077219 20015 init.cc:377] Logged in from keytab as kudu/127.27.153.124@KRBTEST.COM (short username kudu)
I20260810 03:53:10.078190 20015 webserver.cc:460] Webserver started at http://127.27.153.124:45281/ using document root <none> and password file <none>
I20260810 03:53:10.078775 20015 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:10.078841 20015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:10.079052 20015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:53:10.081194 20015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-2/data/instance:
uuid: "ebab8a827b514c4595de3b51a0147f27"
format_stamp: "Formatted at 2026-08-10 03:53:10 on dist-test-slave-pww0"
server_key: "7b880519450c7000094c676365169921"
server_key_iv: "e7220d87d7f87074b8f647ba611175f7"
server_key_version: "encryptionkey@0"
I20260810 03:53:10.081770 20015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-2/wal/instance:
uuid: "ebab8a827b514c4595de3b51a0147f27"
format_stamp: "Formatted at 2026-08-10 03:53:10 on dist-test-slave-pww0"
server_key: "7b880519450c7000094c676365169921"
server_key_iv: "e7220d87d7f87074b8f647ba611175f7"
server_key_version: "encryptionkey@0"
I20260810 03:53:10.085495 20015 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260810 03:53:10.087934 20045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:10.089178 20015 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:53:10.089293 20015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-2/wal
uuid: "ebab8a827b514c4595de3b51a0147f27"
format_stamp: "Formatted at 2026-08-10 03:53:10 on dist-test-slave-pww0"
server_key: "7b880519450c7000094c676365169921"
server_key_iv: "e7220d87d7f87074b8f647ba611175f7"
server_key_version: "encryptionkey@0"
I20260810 03:53:10.089406 20015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:10.098400 20015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:10.101395 20015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:10.101583 20015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:10.108611 20015 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.124:35765
I20260810 03:53:10.108625 20134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.124:35765 every 8 connection(s)
I20260810 03:53:10.109604 20015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-2/data/info.pb
I20260810 03:53:10.110517 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 20015
I20260810 03:53:10.110641 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786333814837068-28261-0/minicluster-data/master-2/wal/instance
I20260810 03:53:10.110965 28261 external_mini_cluster.cc:1428] Setting key 51a22f336f265a2a23664d494f3cb30b
I20260810 03:53:10.113124 20135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:53:10.117602 20135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33531 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:53:10 dist-test-slave-pww0 krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.124@KRBTEST.COM
Aug 10 03:53:10 dist-test-slave-pww0 krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333990, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.124@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:53:10.125651 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:10.112335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35430 (local address 127.27.153.124:35765)
0810 03:53:10.112743 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:10.112763 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:10.112804 (+ 41us) server_negotiation.cc:407] Connection header received
0810 03:53:10.113504 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:10.113525 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:10.113878 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:10.114183 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:10.115119 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.116184 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.117331 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.117799 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.117920 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:10.120873 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:10.120878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:10.120886 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:10.120913 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:10.122478 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:10.122956 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:10.122962 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:10.122968 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:10.123120 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:10.123433 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:10.123436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:10.123437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:10.123944 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:10.124093 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:10.124391 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 03:53:10.124687 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":108,"threads_started":1}
I20260810 03:53:10.129902 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:10.119758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.27.153.126:33531)
0810 03:53:10.119915 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:10.119920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:10.120297 (+ 377us) server_negotiation.cc:407] Connection header received
0810 03:53:10.120527 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:10.120530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:10.120577 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:10.120644 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:10.121371 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.121874 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.122759 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.123007 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.123110 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:10.125552 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:10.125556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:10.125559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:10.125588 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:10.128017 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:10.128586 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:10.128589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:10.128590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:10.128696 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:10.128998 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:10.129001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:10.129003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:10.129257 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:10.129363 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:10.129680 (+ 317us) server_negotiation.cc:299] Negotiation successful
0810 03:53:10.129783 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:53:10.130015 20139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:10.119692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:33531 (local address 127.0.0.1:53454)
0810 03:53:10.120151 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:10.120207 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:10.120408 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:10.120726 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:10.120732 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:10.121099 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:10.121236 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:10.121242 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.121983 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.121986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:10.122632 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:10.122643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.123124 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.123127 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:10.123234 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:10.123815 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:10.123826 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:10.125409 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:10.128206 (+ 2797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:10.128216 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:10.128222 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:10.128490 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:10.128806 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:10.128809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:10.128811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:10.128905 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:10.129352 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:10.129356 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:10.129458 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 03:53:10.129600 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 03:53:10.129810 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":157,"threads_started":1}
I20260810 03:53:10.131108 20135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:53:10 dist-test-slave-pww0 krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
Aug 10 03:53:10 dist-test-slave-pww0 krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333990, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.124@KRBTEST.COM for kudu/127.27.153.125@KRBTEST.COM
I20260810 03:53:10.137655 20140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:10.128229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53458 (local address 127.27.153.126:33531)
0810 03:53:10.128637 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:10.128645 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:10.128656 (+ 11us) server_negotiation.cc:407] Connection header received
0810 03:53:10.128686 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:10.128688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:10.128722 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:10.128805 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:10.129522 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.130283 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.131127 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.131436 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.131562 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:10.134157 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:10.134161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:10.134164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:10.134193 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:10.135927 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:10.136435 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:10.136438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:10.136440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:10.136535 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:10.136777 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:10.136780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:10.136782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:10.136996 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:10.137250 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:10.137325 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 03:53:10.137451 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":141,"threads_started":1}
I20260810 03:53:10.139765 20139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:10.131622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:36189 (local address 127.0.0.1:40150)
0810 03:53:10.131797 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:10.131810 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:10.131881 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:10.132096 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:10.132099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:10.132264 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:10.132386 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:10.132394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.133148 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.133151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:10.133709 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:10.133720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.134266 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.134270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:10.134352 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:10.134882 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:10.134892 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:10.136248 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:10.138092 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:10.138095 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:10.138097 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:10.138341 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:10.138633 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:10.138637 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:10.138639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:10.138735 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:10.139190 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:10.139193 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:10.139306 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 03:53:10.139409 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 03:53:10.139595 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260810 03:53:10.139838 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:10.131728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40150 (local address 127.27.153.125:36189)
0810 03:53:10.131843 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:10.131848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:10.131865 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:53:10.131951 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:10.131954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:10.132004 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:10.132082 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:10.132509 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.133039 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.133846 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.134145 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.134275 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:10.136435 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:10.136441 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:10.136444 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:10.136471 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:10.138006 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:10.138441 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:10.138444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:10.138446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:10.138537 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:10.138867 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:10.138870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:10.138872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:10.139087 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:10.139185 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:10.139564 (+ 379us) server_negotiation.cc:299] Negotiation successful
0810 03:53:10.139693 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 03:53:10.140074 19864 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } attempt: 3
I20260810 03:53:10.140722 20135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:53:10 dist-test-slave-pww0 krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.125@KRBTEST.COM
Aug 10 03:53:10 dist-test-slave-pww0 krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333990, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.124@KRBTEST.COM for kudu/127.27.153.124@KRBTEST.COM
Aug 10 03:53:10 dist-test-slave-pww0 krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333989, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for kudu/127.27.153.124@KRBTEST.COM
I20260810 03:53:10.149325 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:10.138987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40158 (local address 127.27.153.125:36189)
0810 03:53:10.139106 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:10.139110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:10.139160 (+ 50us) server_negotiation.cc:407] Connection header received
0810 03:53:10.139273 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:10.139275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:10.139310 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:10.139365 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:10.140172 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.140867 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.142069 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.142339 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.142443 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:10.145103 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:10.145107 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:10.145109 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:10.145139 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:10.146900 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:10.147398 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:10.147401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:10.147404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:10.147557 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:10.147871 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:10.147874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:10.147876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:10.148062 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:10.148274 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:10.148513 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 03:53:10.148648 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 03:53:10.150029 20002 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } attempt: 3
I20260810 03:53:10.151700 20139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:10.141193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:35765 (local address 127.0.0.1:35442)
0810 03:53:10.141610 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:10.141623 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:10.141688 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:10.142321 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:10.142324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:10.142488 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:10.142575 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:10.142582 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.143559 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.143562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:10.144356 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:10.144366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.145023 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.145026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:10.145097 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:10.145509 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:10.145524 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:10.147626 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:10.149970 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:10.149973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:10.149975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:10.150324 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:10.150603 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:10.150605 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:10.150607 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:10.150684 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:10.151175 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:10.151178 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:10.151263 (+ 85us) client_negotiation.cc:769] Sending connection context
0810 03:53:10.151362 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 03:53:10.151544 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356}
I20260810 03:53:10.151818 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:10.141146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:35765 (local address 127.0.0.1:35438)
0810 03:53:10.141266 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:10.141280 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:10.141357 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:10.141580 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:10.141583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:10.141745 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:10.141828 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:10.141833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.143606 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.143610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:10.144418 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:10.144429 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.145105 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.145108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:10.145211 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:10.145652 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:10.145663 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:10.147238 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:10.150028 (+ 2790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:10.150031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:10.150033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:10.150325 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:10.150745 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:10.150749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:10.150750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:10.150850 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:10.151292 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:10.151294 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:10.151403 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 03:53:10.151514 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 03:53:10.151668 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 03:53:10.152112 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:10.141367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35442 (local address 127.27.153.124:35765)
0810 03:53:10.142102 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:10.142111 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:10.142123 (+ 12us) server_negotiation.cc:407] Connection header received
0810 03:53:10.142164 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:10.142166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:10.142201 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:10.142286 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:10.142677 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.143432 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.144482 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.144915 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.145006 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:10.147936 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:10.147939 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:10.147940 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:10.147962 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:10.149876 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:10.150433 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:10.150436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:10.150437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:10.150519 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:10.150768 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:10.150771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:10.150773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:10.151063 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:10.151145 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:10.151407 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 03:53:10.151948 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":451,"server-negotiator.queue_time_us":672,"thread_start_us":120,"threads_started":1}
I20260810 03:53:10.152889 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:10.141224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35438 (local address 127.27.153.124:35765)
0810 03:53:10.141337 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:10.141341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:10.141352 (+ 11us) server_negotiation.cc:407] Connection header received
0810 03:53:10.141439 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:10.141441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:10.141474 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:10.141545 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:10.141934 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.143426 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.144538 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.144977 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.145401 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:10.147387 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:10.147391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:10.147394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:10.147422 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:10.149923 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:10.150542 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:10.150545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:10.150547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:10.150633 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:10.150962 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:10.150964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:10.150966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:10.151183 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:10.152584 (+ 1401us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:10.152654 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 03:53:10.152760 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":45}
I20260810 03:53:10.153205 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:53:10.155530 20135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebab8a827b514c4595de3b51a0147f27: Bootstrap starting.
I20260810 03:53:10.156037 19864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668: Bootstrap starting.
I20260810 03:53:10.158499 20135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebab8a827b514c4595de3b51a0147f27: Neither blocks nor log segments found. Creating new log.
I20260810 03:53:10.158499 19864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668: Neither blocks nor log segments found. Creating new log.
I20260810 03:53:10.159512 19864 log.cc:824] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:10.159672 20135 log.cc:824] T 00000000000000000000000000000000 P ebab8a827b514c4595de3b51a0147f27: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:10.162045 19864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668: No bootstrap required, opened a new log
I20260810 03:53:10.162387 20135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebab8a827b514c4595de3b51a0147f27: No bootstrap required, opened a new log
I20260810 03:53:10.163579 20012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:10.155341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40174 (local address 127.27.153.125:36189)
0810 03:53:10.155450 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:10.155492 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:10.155731 (+ 239us) server_negotiation.cc:407] Connection header received
0810 03:53:10.155872 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:10.155876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:10.155922 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:10.155985 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:10.156703 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.157201 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.157983 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.158218 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.158306 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:10.159859 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:10.159863 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:10.159866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:10.159896 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:10.161556 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:10.162057 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:10.162061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:10.162063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:10.162139 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:10.162474 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:10.162478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:10.162480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:10.162680 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:10.162760 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:10.163122 (+ 362us) server_negotiation.cc:299] Negotiation successful
0810 03:53:10.163337 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 03:53:10.166404 19864 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc7378be5db47d3b7959f8d29d9c668" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33531 } } peers { permanent_uuid: "1ee62bde015645f1bd0777a3de36d3cb" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } } peers { permanent_uuid: "ebab8a827b514c4595de3b51a0147f27" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } }
I20260810 03:53:10.166405 20135 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebab8a827b514c4595de3b51a0147f27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc7378be5db47d3b7959f8d29d9c668" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33531 } } peers { permanent_uuid: "1ee62bde015645f1bd0777a3de36d3cb" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } } peers { permanent_uuid: "ebab8a827b514c4595de3b51a0147f27" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } }
I20260810 03:53:10.166682 20135 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebab8a827b514c4595de3b51a0147f27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:53:10.166682 19864 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:53:10.166745 20135 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebab8a827b514c4595de3b51a0147f27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebab8a827b514c4595de3b51a0147f27, State: Initialized, Role: FOLLOWER
I20260810 03:53:10.166756 19864 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbc7378be5db47d3b7959f8d29d9c668, State: Initialized, Role: FOLLOWER
I20260810 03:53:10.167302 20135 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebab8a827b514c4595de3b51a0147f27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc7378be5db47d3b7959f8d29d9c668" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33531 } } peers { permanent_uuid: "1ee62bde015645f1bd0777a3de36d3cb" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } } peers { permanent_uuid: "ebab8a827b514c4595de3b51a0147f27" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } }
I20260810 03:53:10.167305 19864 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc7378be5db47d3b7959f8d29d9c668" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33531 } } peers { permanent_uuid: "1ee62bde015645f1bd0777a3de36d3cb" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } } peers { permanent_uuid: "ebab8a827b514c4595de3b51a0147f27" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } }
I20260810 03:53:10.167905 20158 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc7378be5db47d3b7959f8d29d9c668" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33531 } } peers { permanent_uuid: "1ee62bde015645f1bd0777a3de36d3cb" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } } peers { permanent_uuid: "ebab8a827b514c4595de3b51a0147f27" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } } }
I20260810 03:53:10.168035 20158 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668 [sys.catalog]: This master's current role is: FOLLOWER
Aug 10 03:53:10 dist-test-slave-pww0 krb5kdc[19725](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333989, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.125@KRBTEST.COM for kudu/127.27.153.124@KRBTEST.COM
I20260810 03:53:10.168671 20160 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebab8a827b514c4595de3b51a0147f27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc7378be5db47d3b7959f8d29d9c668" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33531 } } peers { permanent_uuid: "1ee62bde015645f1bd0777a3de36d3cb" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } } peers { permanent_uuid: "ebab8a827b514c4595de3b51a0147f27" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } } }
I20260810 03:53:10.168870 20160 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebab8a827b514c4595de3b51a0147f27 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:53:10.168951 19864 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:53:10.169457 20135 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebab8a827b514c4595de3b51a0147f27 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 03:53:10.170219 19786 tablet.cc:2369] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:53:10.173625 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:10.150614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:35765 (local address 127.0.0.1:35444)
0810 03:53:10.150901 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:10.150917 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:10.150993 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:10.151866 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:10.151869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:10.152047 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:10.152157 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:10.152165 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.159975 (+ 7810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.159979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:10.160755 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:10.160768 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.161412 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.161416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:10.161507 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:10.161939 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:10.161952 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:10.168538 (+ 6586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:10.171421 (+ 2883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:10.171426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:10.171428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:10.171777 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:10.172147 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:10.172150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:10.172152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:10.172256 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:10.173103 (+ 847us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:10.173106 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:10.173214 (+ 108us) client_negotiation.cc:769] Sending connection context
0810 03:53:10.173297 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 03:53:10.173456 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179}
I20260810 03:53:10.174554 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:10.151125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35444 (local address 127.27.153.124:35765)
0810 03:53:10.151629 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:10.151637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:10.151649 (+ 12us) server_negotiation.cc:407] Connection header received
0810 03:53:10.151679 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:10.151682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:10.151721 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:10.152267 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:10.152888 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.159856 (+ 6968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.160906 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.161268 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.161706 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:10.168780 (+ 7074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:10.168784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:10.168787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:10.168860 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:10.171286 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:10.171908 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:10.171912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:10.171913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:10.172014 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:10.172379 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:10.172382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:10.172384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:10.172984 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:10.174180 (+ 1196us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:10.174254 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 03:53:10.174388 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":107,"threads_started":1}
W20260810 03:53:10.176100 20198 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:53:10.176177 20198 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:53:10.177706 20002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb: Bootstrap starting.
I20260810 03:53:10.176231 19748 master_runner.cc:186] Error getting master registration for 127.27.153.125:36189: OK, instance_id { permanent_uuid: "1ee62bde015645f1bd0777a3de36d3cb" instance_seqno: 1786333989896525 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 03:53:10.179694 19748 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 03:53:10.180472 20002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb: Neither blocks nor log segments found. Creating new log.
W20260810 03:53:10.180804 20201 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ebab8a827b514c4595de3b51a0147f27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:53:10.180891 20201 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:53:10.181439 20002 log.cc:824] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:10.183976 20140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:10.173626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53472 (local address 127.27.153.126:33531)
0810 03:53:10.173920 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:10.173924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:10.173935 (+ 11us) server_negotiation.cc:407] Connection header received
0810 03:53:10.173967 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:10.173970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:10.174011 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:10.174075 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:10.175159 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.175927 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.177129 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.177482 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.177576 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:10.179399 (+ 1823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:10.179403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:10.179406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:10.179435 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:10.181437 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:10.182059 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:10.182063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:10.182065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:10.182166 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:10.182529 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:10.182533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:10.182535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:10.182777 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:10.182924 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:10.183662 (+ 738us) server_negotiation.cc:299] Negotiation successful
0810 03:53:10.183811 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260810 03:53:10.183368 20015 master_runner.cc:186] Error getting master registration for 127.27.153.125:36189: OK, instance_id { permanent_uuid: "1ee62bde015645f1bd0777a3de36d3cb" instance_seqno: 1786333989896525 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 03:53:10.186730 20015 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 03:53:10.189772 20002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb: No bootstrap required, opened a new log
I20260810 03:53:10.191411 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:10.173437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35458 (local address 127.27.153.124:35765)
0810 03:53:10.173546 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:10.173550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:10.182265 (+ 8715us) server_negotiation.cc:407] Connection header received
0810 03:53:10.182467 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:10.182470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:10.182517 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:10.182566 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:10.183793 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.184539 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.185665 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:10.185988 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:10.186062 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:10.187886 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:10.187890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:10.187893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:10.187919 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:10.189469 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:10.190009 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:10.190013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:10.190015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:10.190107 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:10.190457 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:10.190460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:10.190462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:10.190705 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:10.190831 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:10.191124 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 03:53:10.191249 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 03:53:10.193666 20002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc7378be5db47d3b7959f8d29d9c668" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33531 } } peers { permanent_uuid: "1ee62bde015645f1bd0777a3de36d3cb" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } } peers { permanent_uuid: "ebab8a827b514c4595de3b51a0147f27" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } }
I20260810 03:53:10.193898 20002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:53:10.193958 20002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ee62bde015645f1bd0777a3de36d3cb, State: Initialized, Role: FOLLOWER
I20260810 03:53:10.194445 20002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc7378be5db47d3b7959f8d29d9c668" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33531 } } peers { permanent_uuid: "1ee62bde015645f1bd0777a3de36d3cb" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } } peers { permanent_uuid: "ebab8a827b514c4595de3b51a0147f27" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } }
I20260810 03:53:10.194926 20216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc7378be5db47d3b7959f8d29d9c668" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33531 } } peers { permanent_uuid: "1ee62bde015645f1bd0777a3de36d3cb" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } } peers { permanent_uuid: "ebab8a827b514c4595de3b51a0147f27" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } } }
I20260810 03:53:10.195889 20216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:53:10.196722 20002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [sys.catalog]: configured and running, proceeding with master startup.
W20260810 03:53:10.201923 20233 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:53:10.202032 20233 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:53:10.206115 19881 master_runner.cc:263] No leader master found from master 1ee62bde015645f1bd0777a3de36d3cb
I20260810 03:53:10.206199 19881 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 03:53:10.256716 20216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:53:10.256945 20216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc7378be5db47d3b7959f8d29d9c668" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33531 } } peers { permanent_uuid: "1ee62bde015645f1bd0777a3de36d3cb" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } } peers { permanent_uuid: "ebab8a827b514c4595de3b51a0147f27" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } }
I20260810 03:53:10.258117 20216 leader_election.cc:288] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbc7378be5db47d3b7959f8d29d9c668 (127.27.153.126:33531), ebab8a827b514c4595de3b51a0147f27 (127.27.153.124:35765)
I20260810 03:53:10.259351 20089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ee62bde015645f1bd0777a3de36d3cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebab8a827b514c4595de3b51a0147f27" is_pre_election: true
I20260810 03:53:10.259761 20089 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ebab8a827b514c4595de3b51a0147f27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ee62bde015645f1bd0777a3de36d3cb in term 0.
I20260810 03:53:10.260608 19819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ee62bde015645f1bd0777a3de36d3cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbc7378be5db47d3b7959f8d29d9c668" is_pre_election: true
I20260810 03:53:10.260972 19819 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ee62bde015645f1bd0777a3de36d3cb in term 0.
I20260810 03:53:10.261952 19907 leader_election.cc:302] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ee62bde015645f1bd0777a3de36d3cb, bbc7378be5db47d3b7959f8d29d9c668; no voters:
I20260810 03:53:10.262466 20216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:53:10.262566 20216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:53:10.262615 20216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:53:10.264564 20216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc7378be5db47d3b7959f8d29d9c668" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33531 } } peers { permanent_uuid: "1ee62bde015645f1bd0777a3de36d3cb" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } } peers { permanent_uuid: "ebab8a827b514c4595de3b51a0147f27" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } }
I20260810 03:53:10.264950 20216 leader_election.cc:288] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [CANDIDATE]: Term 1 election: Requested vote from peers bbc7378be5db47d3b7959f8d29d9c668 (127.27.153.126:33531), ebab8a827b514c4595de3b51a0147f27 (127.27.153.124:35765)
I20260810 03:53:10.265843 19819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ee62bde015645f1bd0777a3de36d3cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbc7378be5db47d3b7959f8d29d9c668"
I20260810 03:53:10.265986 19819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:53:10.267355 19819 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ee62bde015645f1bd0777a3de36d3cb in term 1.
I20260810 03:53:10.267817 20089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1ee62bde015645f1bd0777a3de36d3cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebab8a827b514c4595de3b51a0147f27"
I20260810 03:53:10.267961 20089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebab8a827b514c4595de3b51a0147f27 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:53:10.269512 20089 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ebab8a827b514c4595de3b51a0147f27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ee62bde015645f1bd0777a3de36d3cb in term 1.
I20260810 03:53:10.269948 19907 leader_election.cc:302] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ee62bde015645f1bd0777a3de36d3cb, ebab8a827b514c4595de3b51a0147f27; no voters:
I20260810 03:53:10.271821 20216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:53:10.272087 20216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [term 1 LEADER]: Becoming Leader. State: Replica: 1ee62bde015645f1bd0777a3de36d3cb, State: Running, Role: LEADER
I20260810 03:53:10.272457 20216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc7378be5db47d3b7959f8d29d9c668" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33531 } } peers { permanent_uuid: "1ee62bde015645f1bd0777a3de36d3cb" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } } peers { permanent_uuid: "ebab8a827b514c4595de3b51a0147f27" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } }
I20260810 03:53:10.274596 20216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ee62bde015645f1bd0777a3de36d3cb. Latest consensus state: current_term: 1 leader_uuid: "1ee62bde015645f1bd0777a3de36d3cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc7378be5db47d3b7959f8d29d9c668" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33531 } } peers { permanent_uuid: "1ee62bde015645f1bd0777a3de36d3cb" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } } peers { permanent_uuid: "ebab8a827b514c4595de3b51a0147f27" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } } }
I20260810 03:53:10.274725 20216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [sys.catalog]: This master's current role is: LEADER
I20260810 03:53:10.276861 20238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:53:10.277976 20238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:53:10.287007 20089 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ebab8a827b514c4595de3b51a0147f27 [term 1 FOLLOWER]: Refusing update from remote peer 1ee62bde015645f1bd0777a3de36d3cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:53:10.287808 20236 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebab8a827b514c4595de3b51a0147f27" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:53:10.291992 20160 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebab8a827b514c4595de3b51a0147f27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ee62bde015645f1bd0777a3de36d3cb. Latest consensus state: current_term: 1 leader_uuid: "1ee62bde015645f1bd0777a3de36d3cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc7378be5db47d3b7959f8d29d9c668" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33531 } } peers { permanent_uuid: "1ee62bde015645f1bd0777a3de36d3cb" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } } peers { permanent_uuid: "ebab8a827b514c4595de3b51a0147f27" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } } }
I20260810 03:53:10.292158 20160 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebab8a827b514c4595de3b51a0147f27 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:53:10.293435 19819 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668 [term 1 FOLLOWER]: Refusing update from remote peer 1ee62bde015645f1bd0777a3de36d3cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:53:10.296752 20236 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbc7378be5db47d3b7959f8d29d9c668" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:53:10.301206 20158 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ee62bde015645f1bd0777a3de36d3cb. Latest consensus state: current_term: 1 leader_uuid: "1ee62bde015645f1bd0777a3de36d3cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc7378be5db47d3b7959f8d29d9c668" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33531 } } peers { permanent_uuid: "1ee62bde015645f1bd0777a3de36d3cb" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } } peers { permanent_uuid: "ebab8a827b514c4595de3b51a0147f27" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } } }
I20260810 03:53:10.301373 20158 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:53:10.311488 20236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ee62bde015645f1bd0777a3de36d3cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc7378be5db47d3b7959f8d29d9c668" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33531 } } peers { permanent_uuid: "1ee62bde015645f1bd0777a3de36d3cb" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } } peers { permanent_uuid: "ebab8a827b514c4595de3b51a0147f27" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } } }
I20260810 03:53:10.311671 20236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [sys.catalog]: This master's current role is: LEADER
I20260810 03:53:10.313776 20158 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ee62bde015645f1bd0777a3de36d3cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc7378be5db47d3b7959f8d29d9c668" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33531 } } peers { permanent_uuid: "1ee62bde015645f1bd0777a3de36d3cb" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } } peers { permanent_uuid: "ebab8a827b514c4595de3b51a0147f27" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } } }
I20260810 03:53:10.313949 20158 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:53:10.314435 20239 mvcc.cc:205] Tried to move back new op lower bound from 7316824024206831616 to 7316824024160395264. Current Snapshot: MvccSnapshot[applied={T|T < 7316824024206831616}]
I20260810 03:53:10.314936 20242 mvcc.cc:205] Tried to move back new op lower bound from 7316824024206831616 to 7316824024160395264. Current Snapshot: MvccSnapshot[applied={T|T < 7316824024206831616}]
I20260810 03:53:10.317958 20216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1ee62bde015645f1bd0777a3de36d3cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc7378be5db47d3b7959f8d29d9c668" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33531 } } peers { permanent_uuid: "1ee62bde015645f1bd0777a3de36d3cb" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } } peers { permanent_uuid: "ebab8a827b514c4595de3b51a0147f27" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } } }
I20260810 03:53:10.318106 20216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb [sys.catalog]: This master's current role is: LEADER
I20260810 03:53:10.319511 20160 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebab8a827b514c4595de3b51a0147f27 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1ee62bde015645f1bd0777a3de36d3cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbc7378be5db47d3b7959f8d29d9c668" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33531 } } peers { permanent_uuid: "1ee62bde015645f1bd0777a3de36d3cb" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 36189 } } peers { permanent_uuid: "ebab8a827b514c4595de3b51a0147f27" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 35765 } } }
I20260810 03:53:10.319660 20160 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebab8a827b514c4595de3b51a0147f27 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:53:10.323557 20240 mvcc.cc:205] Tried to move back new op lower bound from 7316824024206831616 to 7316824024160395264. Current Snapshot: MvccSnapshot[applied={T|T < 7316824024206831616}]
I20260810 03:53:10.328487 20238 catalog_manager.cc:1347] Generated new cluster ID: 1b849d12e7304f8eb5c8ba2a8699dbb4
I20260810 03:53:10.328599 20238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:53:10.358604 20238 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:53:10.360087 20238 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:53:10.376436 20238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ee62bde015645f1bd0777a3de36d3cb: Generated new TSK 0
I20260810 03:53:10.377238 20238 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:53:11.177577 20198 catalog_manager.cc:1259] Loaded cluster ID: 1b849d12e7304f8eb5c8ba2a8699dbb4
I20260810 03:53:11.177665 20198 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668: loading cluster ID for follower catalog manager: success
I20260810 03:53:11.179654 20198 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668: acquiring CA information for follower catalog manager: success
I20260810 03:53:11.180927 20198 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bbc7378be5db47d3b7959f8d29d9c668: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:53:11.182037 20201 catalog_manager.cc:1259] Loaded cluster ID: 1b849d12e7304f8eb5c8ba2a8699dbb4
I20260810 03:53:11.182099 20201 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ebab8a827b514c4595de3b51a0147f27: loading cluster ID for follower catalog manager: success
I20260810 03:53:11.183678 20201 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ebab8a827b514c4595de3b51a0147f27: acquiring CA information for follower catalog manager: success
I20260810 03:53:11.184801 20201 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ebab8a827b514c4595de3b51a0147f27: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:53:11.456231 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 19748
I20260810 03:53:11.476581 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 19881
I20260810 03:53:11.497529 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 20015
2026-08-10T03:53:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3368 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:53:11 dist-test-slave-pww0 krb5kdc[20259](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:53:11 dist-test-slave-pww0 krb5kdc[20259](info): set up 2 sockets
Aug 10 03:53:11 dist-test-slave-pww0 krb5kdc[20259](info): commencing operation
krb5kdc: starting...
W20260810 03:53:12.708298 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:53:12 dist-test-slave-pww0 krb5kdc[20259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:53:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:53:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:53:12.838220 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:41025
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:40353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:41025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:53:12.952451 20279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:12.952735 20279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:12.952776 20279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:12.956836 20279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:53:12.956905 20279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:12.956923 20279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:12.956940 20279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:53:12.956956 20279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:53:12.962478 20279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:41025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:41025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:12.963680 20279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:12.964752 20279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:12.973556 20285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:12.973560 20287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:12.973560 20284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:12.973619 20279 server_base.cc:1034] running on GCE node
I20260810 03:53:12.974485 20279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:12.975782 20279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:12.976944 20279 hybrid_clock.cc:648] HybridClock initialized: now 1786333992976918 us; error 48 us; skew 500 ppm
Aug 10 03:53:12 dist-test-slave-pww0 krb5kdc[20259](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333992, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:12.979839 20279 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:53:12.980824 20279 webserver.cc:460] Webserver started at http://127.27.153.126:36055/ using document root <none> and password file <none>
I20260810 03:53:12.981408 20279 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:12.981478 20279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:12.981683 20279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:53:12.983881 20279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "53ec35f803d64e13ba852ba17f4760bb"
format_stamp: "Formatted at 2026-08-10 03:53:12 on dist-test-slave-pww0"
server_key: "fb25cc59923483a839abca7e302d1e50"
server_key_iv: "f1e627fc7f8240b16e954954da1ebfd9"
server_key_version: "encryptionkey@0"
I20260810 03:53:12.984539 20279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "53ec35f803d64e13ba852ba17f4760bb"
format_stamp: "Formatted at 2026-08-10 03:53:12 on dist-test-slave-pww0"
server_key: "fb25cc59923483a839abca7e302d1e50"
server_key_iv: "f1e627fc7f8240b16e954954da1ebfd9"
server_key_version: "encryptionkey@0"
I20260810 03:53:12.988657 20279 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:53:12.991549 20298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:12.992976 20279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:53:12.993131 20279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "53ec35f803d64e13ba852ba17f4760bb"
format_stamp: "Formatted at 2026-08-10 03:53:12 on dist-test-slave-pww0"
server_key: "fb25cc59923483a839abca7e302d1e50"
server_key_iv: "f1e627fc7f8240b16e954954da1ebfd9"
server_key_version: "encryptionkey@0"
I20260810 03:53:12.993280 20279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:13.023100 20279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:13.026311 20279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:13.026516 20279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:13.034053 20279 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:41025
I20260810 03:53:13.034077 20411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:41025 every 8 connection(s)
I20260810 03:53:13.035095 20279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:53:13.036015 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 20279
I20260810 03:53:13.036154 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:53:13.036487 28261 external_mini_cluster.cc:1428] Setting key d10fe673b81ea9821381e0541a07347a
I20260810 03:53:13.038688 20412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 03:53:13 dist-test-slave-pww0 krb5kdc[20259](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:53:13.046036 20412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53ec35f803d64e13ba852ba17f4760bb: Bootstrap starting.
I20260810 03:53:13.048864 20412 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53ec35f803d64e13ba852ba17f4760bb: Neither blocks nor log segments found. Creating new log.
I20260810 03:53:13.049801 20412 log.cc:824] T 00000000000000000000000000000000 P 53ec35f803d64e13ba852ba17f4760bb: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:13.051667 20421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:13.037895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50982 (local address 127.27.153.126:41025)
0810 03:53:13.038323 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:13.038340 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:13.038380 (+ 40us) server_negotiation.cc:407] Connection header received
0810 03:53:13.039076 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:13.039097 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:13.039489 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:13.039804 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:13.040749 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:13.041781 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:13.042973 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:13.043491 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:13.043614 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:13.046685 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:13.046691 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:13.046707 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:13.046742 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:13.048465 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:13.049038 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:13.049044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:13.049051 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:13.049197 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:13.049496 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:13.049498 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:13.049500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:13.049971 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:13.050117 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:13.050516 (+ 399us) server_negotiation.cc:299] Negotiation successful
0810 03:53:13.050837 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":114,"threads_started":1}
I20260810 03:53:13.052194 20412 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53ec35f803d64e13ba852ba17f4760bb: No bootstrap required, opened a new log
I20260810 03:53:13.054814 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:53:13.056906 20412 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53ec35f803d64e13ba852ba17f4760bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ec35f803d64e13ba852ba17f4760bb" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41025 } }
I20260810 03:53:13.057166 20412 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53ec35f803d64e13ba852ba17f4760bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:53:13.057224 20412 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53ec35f803d64e13ba852ba17f4760bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53ec35f803d64e13ba852ba17f4760bb, State: Initialized, Role: FOLLOWER
I20260810 03:53:13.057626 20412 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53ec35f803d64e13ba852ba17f4760bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ec35f803d64e13ba852ba17f4760bb" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41025 } }
I20260810 03:53:13.057746 20412 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53ec35f803d64e13ba852ba17f4760bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:53:13.057794 20412 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53ec35f803d64e13ba852ba17f4760bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:53:13.057869 20412 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53ec35f803d64e13ba852ba17f4760bb [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:53:13.058874 20412 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53ec35f803d64e13ba852ba17f4760bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ec35f803d64e13ba852ba17f4760bb" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41025 } }
I20260810 03:53:13.059221 20412 leader_election.cc:302] T 00000000000000000000000000000000 P 53ec35f803d64e13ba852ba17f4760bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53ec35f803d64e13ba852ba17f4760bb; no voters:
I20260810 03:53:13.059557 20412 leader_election.cc:288] T 00000000000000000000000000000000 P 53ec35f803d64e13ba852ba17f4760bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:53:13.059679 20423 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53ec35f803d64e13ba852ba17f4760bb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:53:13.059926 20423 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53ec35f803d64e13ba852ba17f4760bb [term 1 LEADER]: Becoming Leader. State: Replica: 53ec35f803d64e13ba852ba17f4760bb, State: Running, Role: LEADER
I20260810 03:53:13.060283 20423 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53ec35f803d64e13ba852ba17f4760bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ec35f803d64e13ba852ba17f4760bb" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41025 } }
I20260810 03:53:13.060856 20412 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53ec35f803d64e13ba852ba17f4760bb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:53:13.061970 20426 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ec35f803d64e13ba852ba17f4760bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53ec35f803d64e13ba852ba17f4760bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ec35f803d64e13ba852ba17f4760bb" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41025 } } }
I20260810 03:53:13.062112 20426 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ec35f803d64e13ba852ba17f4760bb [sys.catalog]: This master's current role is: LEADER
I20260810 03:53:13.062464 20428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53ec35f803d64e13ba852ba17f4760bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53ec35f803d64e13ba852ba17f4760bb. Latest consensus state: current_term: 1 leader_uuid: "53ec35f803d64e13ba852ba17f4760bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53ec35f803d64e13ba852ba17f4760bb" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 41025 } } }
I20260810 03:53:13.062606 20428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53ec35f803d64e13ba852ba17f4760bb [sys.catalog]: This master's current role is: LEADER
I20260810 03:53:13.062515 20435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:53:13.066147 20435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:53:13.072530 20435 catalog_manager.cc:1347] Generated new cluster ID: 5ef20d398e0f4f4ca7dfe4d4b362b027
I20260810 03:53:13.072631 20435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:53:13.095955 20435 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:53:13.097085 20435 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:53:13.104890 20435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53ec35f803d64e13ba852ba17f4760bb: Generated new TSK 0
I20260810 03:53:13.105700 20435 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:53:13.111289 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 20279
2026-08-10T03:53:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1638 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:53:13 dist-test-slave-pww0 krb5kdc[20456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:53:13 dist-test-slave-pww0 krb5kdc[20456](info): set up 2 sockets
Aug 10 03:53:13 dist-test-slave-pww0 krb5kdc[20456](info): commencing operation
krb5kdc: starting...
W20260810 03:53:14.343255 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:53:14 dist-test-slave-pww0 krb5kdc[20456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:53:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:53:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:53:14.472182 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:45951
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:45951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:53:14.587384 20480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:14.587728 20480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:14.587798 20480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:14.591946 20480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:53:14.592021 20480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:14.592047 20480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:14.592069 20480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:53:14.592090 20480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:53:14.597729 20480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:45951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:45951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:14.598953 20480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:14.600123 20480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:14.609151 20493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:14.609232 20480 server_base.cc:1034] running on GCE node
W20260810 03:53:14.609166 20489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:14.609169 20495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:14.610062 20480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:14.611290 20480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:14.612458 20480 hybrid_clock.cc:648] HybridClock initialized: now 1786333994612456 us; error 56 us; skew 500 ppm
Aug 10 03:53:14 dist-test-slave-pww0 krb5kdc[20456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333994, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:14.615520 20480 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:53:14.616533 20480 webserver.cc:460] Webserver started at http://127.27.153.126:34513/ using document root <none> and password file <none>
I20260810 03:53:14.617100 20480 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:14.617192 20480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:14.617415 20480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:53:14.619827 20480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "6709cbea73ae4017848debfebb08f24c"
format_stamp: "Formatted at 2026-08-10 03:53:14 on dist-test-slave-pww0"
server_key: "9f89fede1da93d8154db3a2b3afd1c31"
server_key_iv: "0d1fabdae75bbd2557a31505da525274"
server_key_version: "encryptionkey@0"
I20260810 03:53:14.620396 20480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "6709cbea73ae4017848debfebb08f24c"
format_stamp: "Formatted at 2026-08-10 03:53:14 on dist-test-slave-pww0"
server_key: "9f89fede1da93d8154db3a2b3afd1c31"
server_key_iv: "0d1fabdae75bbd2557a31505da525274"
server_key_version: "encryptionkey@0"
I20260810 03:53:14.624437 20480 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:53:14.627143 20508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:14.628500 20480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 03:53:14.628636 20480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "6709cbea73ae4017848debfebb08f24c"
format_stamp: "Formatted at 2026-08-10 03:53:14 on dist-test-slave-pww0"
server_key: "9f89fede1da93d8154db3a2b3afd1c31"
server_key_iv: "0d1fabdae75bbd2557a31505da525274"
server_key_version: "encryptionkey@0"
I20260810 03:53:14.628756 20480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:14.643738 20480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:14.647150 20480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:14.647369 20480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:14.654888 20480 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:45951
I20260810 03:53:14.655215 20606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:45951 every 8 connection(s)
I20260810 03:53:14.656010 20480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:53:14.658953 20609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:53:14.660128 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 20480
I20260810 03:53:14.660251 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:53:14.660552 28261 external_mini_cluster.cc:1428] Setting key b5a3d4f4378317ab7ef1100110d7361b
I20260810 03:53:14.665452 20609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c: Bootstrap starting.
I20260810 03:53:14.668249 20609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c: Neither blocks nor log segments found. Creating new log.
I20260810 03:53:14.669224 20609 log.cc:824] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:53:14 dist-test-slave-pww0 krb5kdc[20456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:53:14.671286 20609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c: No bootstrap required, opened a new log
I20260810 03:53:14.674790 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:14.661943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.27.153.126:45951)
0810 03:53:14.662364 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:14.662381 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:14.662418 (+ 37us) server_negotiation.cc:407] Connection header received
0810 03:53:14.663071 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:14.663087 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:14.663427 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:14.663764 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:14.664632 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:14.665643 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:14.666781 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:14.667244 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:14.667356 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:14.670142 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:14.670148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:14.670162 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:14.670189 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:14.671925 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:14.672417 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:14.672422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:14.672427 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:14.672543 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:14.672833 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:14.672836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:14.672837 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:14.673229 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:14.673384 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:14.673684 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 03:53:14.673989 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":114,"threads_started":1}
I20260810 03:53:14.675339 20609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709cbea73ae4017848debfebb08f24c" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45951 } }
I20260810 03:53:14.675609 20609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:53:14.675668 20609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6709cbea73ae4017848debfebb08f24c, State: Initialized, Role: FOLLOWER
I20260810 03:53:14.676158 20609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709cbea73ae4017848debfebb08f24c" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45951 } }
I20260810 03:53:14.676317 20609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:53:14.676383 20609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:53:14.676481 20609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:53:14.677534 20609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709cbea73ae4017848debfebb08f24c" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45951 } }
I20260810 03:53:14.677841 20609 leader_election.cc:302] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6709cbea73ae4017848debfebb08f24c; no voters:
I20260810 03:53:14.678112 20609 leader_election.cc:288] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:53:14.678242 20618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:53:14.678458 20618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 1 LEADER]: Becoming Leader. State: Replica: 6709cbea73ae4017848debfebb08f24c, State: Running, Role: LEADER
I20260810 03:53:14.678767 20618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709cbea73ae4017848debfebb08f24c" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45951 } }
I20260810 03:53:14.679328 20609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:53:14.680532 20619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6709cbea73ae4017848debfebb08f24c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709cbea73ae4017848debfebb08f24c" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45951 } } }
I20260810 03:53:14.680547 20620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6709cbea73ae4017848debfebb08f24c. Latest consensus state: current_term: 1 leader_uuid: "6709cbea73ae4017848debfebb08f24c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709cbea73ae4017848debfebb08f24c" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45951 } } }
I20260810 03:53:14.680645 20619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [sys.catalog]: This master's current role is: LEADER
I20260810 03:53:14.680691 20620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [sys.catalog]: This master's current role is: LEADER
I20260810 03:53:14.681356 20627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:53:14.684243 20627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:53:14.689471 20627 catalog_manager.cc:1347] Generated new cluster ID: 20365940fafd4bd18913a6405bb087c3
I20260810 03:53:14.689549 20627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:53:14.702060 20627 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:53:14.703039 20627 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:53:14.708536 20627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c: Generated new TSK 0
I20260810 03:53:14.709257 20627 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.153.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
I20260810 03:53:14.741731 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:45951
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:53:14.861202 20657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:14.861515 20657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:14.861574 20657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:14.865724 20657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:14.865813 20657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:14.865875 20657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:53:14.871393 20657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:45951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:14.872584 20657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:14.873654 20657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:14.883934 20672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:14.883957 20667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:14.883957 20668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:14.884305 20657 server_base.cc:1034] running on GCE node
I20260810 03:53:14.884786 20657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:14.885532 20657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:14.886703 20657 hybrid_clock.cc:648] HybridClock initialized: now 1786333994886677 us; error 39 us; skew 500 ppm
Aug 10 03:53:14 dist-test-slave-pww0 krb5kdc[20456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333994, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:14.889628 20657 init.cc:377] Logged in from keytab as kudu/127.27.153.65@KRBTEST.COM (short username kudu)
I20260810 03:53:14.890635 20657 webserver.cc:460] Webserver started at http://127.27.153.65:38537/ using document root <none> and password file <none>
I20260810 03:53:14.891211 20657 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:14.891284 20657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:14.891542 20657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:53:14.893782 20657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "f28d5e87eeca486f9ab671c30f7e136f"
format_stamp: "Formatted at 2026-08-10 03:53:14 on dist-test-slave-pww0"
server_key: "6c9c1f5ba883dad47917d6f5eaddc93d"
server_key_iv: "9c78c69df8871b6c9c0ea1bfc112b776"
server_key_version: "encryptionkey@0"
I20260810 03:53:14.894381 20657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "f28d5e87eeca486f9ab671c30f7e136f"
format_stamp: "Formatted at 2026-08-10 03:53:14 on dist-test-slave-pww0"
server_key: "6c9c1f5ba883dad47917d6f5eaddc93d"
server_key_iv: "9c78c69df8871b6c9c0ea1bfc112b776"
server_key_version: "encryptionkey@0"
I20260810 03:53:14.898424 20657 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:53:14.901132 20680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:14.902468 20657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260810 03:53:14.902585 20657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "f28d5e87eeca486f9ab671c30f7e136f"
format_stamp: "Formatted at 2026-08-10 03:53:14 on dist-test-slave-pww0"
server_key: "6c9c1f5ba883dad47917d6f5eaddc93d"
server_key_iv: "9c78c69df8871b6c9c0ea1bfc112b776"
server_key_version: "encryptionkey@0"
I20260810 03:53:14.902704 20657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:14.919304 20657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:14.922539 20657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:14.922731 20657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:14.923357 20657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:53:14.924245 20657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:53:14.924293 20657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:14.924340 20657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:53:14.924365 20657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:14.932895 20657 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:38881
I20260810 03:53:14.932931 20890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:38881 every 8 connection(s)
I20260810 03:53:14.933841 20657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:53:14.939724 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 20657
I20260810 03:53:14.939859 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:53:14.940151 28261 external_mini_cluster.cc:1428] Setting key 46b6357182a9f0fe533dfcdfc0f7e317
Aug 10 03:53:14 dist-test-slave-pww0 krb5kdc[20456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333994, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:53:14.948705 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:14.935696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:33927 (local address 127.27.153.126:45951)
0810 03:53:14.935825 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:14.935828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:14.936742 (+ 914us) server_negotiation.cc:407] Connection header received
0810 03:53:14.938109 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:14.938112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:14.938164 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:14.938240 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:14.939442 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:14.940040 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:14.941509 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:14.941805 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:14.941886 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:14.944803 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:14.944807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:14.944810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:14.944841 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:14.946177 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:14.946793 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:14.946796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:14.946797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:14.946867 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:14.947317 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:14.947321 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:14.947323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:14.947615 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:14.947714 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:14.948450 (+ 736us) server_negotiation.cc:299] Negotiation successful
0810 03:53:14.948572 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 03:53:14.949976 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:14.936102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.65:33927)
0810 03:53:14.936577 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:14.936623 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:14.937886 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:14.938411 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:14.938418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:14.938796 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:14.939313 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:14.939329 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:14.940178 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:14.940182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:14.941392 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:14.941405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:14.941956 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:14.941961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:14.942174 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:14.942808 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:14.942830 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:14.944592 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:14.946302 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:14.946309 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:14.946327 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:14.946689 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:14.946951 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:14.946954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:14.946956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:14.947166 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:14.947735 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:14.947742 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:14.948128 (+ 386us) client_negotiation.cc:769] Sending connection context
0810 03:53:14.948379 (+ 251us) client_negotiation.cc:240] Negotiation successful
0810 03:53:14.948798 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":121,"threads_started":1}
I20260810 03:53:14.951372 20892 heartbeater.cc:347] Connected to a master server at 127.27.153.126:45951
WARNING: no policy specified for kudu/127.27.153.66@KRBTEST.COM; defaulting to no policy
I20260810 03:53:14.951676 20892 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:14.952252 20892 heartbeater.cc:511] Master 127.27.153.126:45951 requested a full tablet report, sending...
I20260810 03:53:14.953918 20546 ts_manager.cc:194] Registered new tserver with Master: f28d5e87eeca486f9ab671c30f7e136f (127.27.153.65:38881)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.66@KRBTEST.COM" created.
I20260810 03:53:14.955253 20546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:33927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
I20260810 03:53:14.970855 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:45951
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:53:15.102103 20906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:15.102394 20906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:15.102437 20906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:15.106613 20906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:15.106698 20906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:15.106774 20906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:53:15.112623 20906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:45951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:15.113862 20906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:15.114977 20906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:15.125452 20914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:15.125473 20917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:15.125471 20922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:15.126088 20906 server_base.cc:1034] running on GCE node
I20260810 03:53:15.126641 20906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:15.127357 20906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:15.128481 20906 hybrid_clock.cc:648] HybridClock initialized: now 1786333995128465 us; error 39 us; skew 500 ppm
Aug 10 03:53:15 dist-test-slave-pww0 krb5kdc[20456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333995, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:15.131909 20906 init.cc:377] Logged in from keytab as kudu/127.27.153.66@KRBTEST.COM (short username kudu)
I20260810 03:53:15.132930 20906 webserver.cc:460] Webserver started at http://127.27.153.66:38735/ using document root <none> and password file <none>
I20260810 03:53:15.133509 20906 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:15.133574 20906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:15.133755 20906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:53:15.136013 20906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "d020d94eb8224169bc14c10f564817f0"
format_stamp: "Formatted at 2026-08-10 03:53:15 on dist-test-slave-pww0"
server_key: "c720c48cd55f9e3224da960677eb8b1f"
server_key_iv: "c8853d6c6ae3ab4d2c44050b8cddf8a5"
server_key_version: "encryptionkey@0"
I20260810 03:53:15.136569 20906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "d020d94eb8224169bc14c10f564817f0"
format_stamp: "Formatted at 2026-08-10 03:53:15 on dist-test-slave-pww0"
server_key: "c720c48cd55f9e3224da960677eb8b1f"
server_key_iv: "c8853d6c6ae3ab4d2c44050b8cddf8a5"
server_key_version: "encryptionkey@0"
I20260810 03:53:15.140555 20906 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:53:15.143187 20933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:15.144697 20906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:53:15.144832 20906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "d020d94eb8224169bc14c10f564817f0"
format_stamp: "Formatted at 2026-08-10 03:53:15 on dist-test-slave-pww0"
server_key: "c720c48cd55f9e3224da960677eb8b1f"
server_key_iv: "c8853d6c6ae3ab4d2c44050b8cddf8a5"
server_key_version: "encryptionkey@0"
I20260810 03:53:15.144936 20906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:15.162552 20906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:15.165968 20906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:15.166170 20906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:15.168541 20906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:53:15.169781 20906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:53:15.169842 20906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:15.169886 20906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:53:15.169900 20906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:15.178915 20906 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:39071
I20260810 03:53:15.178944 21147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:39071 every 8 connection(s)
I20260810 03:53:15.180016 20906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:53:15.190224 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 20906
Aug 10 03:53:15 dist-test-slave-pww0 krb5kdc[20456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333995, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:53:15.190335 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:53:15.190636 28261 external_mini_cluster.cc:1428] Setting key ed0aeea6ff75b4180ef0bc2c5dc1a135
I20260810 03:53:15.195528 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:15.182015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:51461 (local address 127.27.153.126:45951)
0810 03:53:15.182141 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:15.182144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:15.183080 (+ 936us) server_negotiation.cc:407] Connection header received
0810 03:53:15.184523 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:15.184526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:15.184580 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:15.184672 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:15.185990 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.186540 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.187867 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.188170 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.188285 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:15.191081 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:15.191084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:15.191087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:15.191120 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:15.192801 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:15.193491 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:15.193495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:15.193496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:15.193606 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:15.194034 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:15.194037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:15.194038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:15.194269 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:15.194358 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:15.195131 (+ 773us) server_negotiation.cc:299] Negotiation successful
0810 03:53:15.195297 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 03:53:15.196337 21157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:15.182431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.66:51461)
0810 03:53:15.182912 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:15.182960 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:15.184272 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:15.184876 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:15.184884 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:15.185289 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:15.185823 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:15.185838 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.186691 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.186694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:15.187725 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:15.187737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.188355 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.188358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:15.188573 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:15.189149 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:15.189173 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:15.190849 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:15.192936 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:15.192943 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:15.192962 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:15.193361 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:15.193704 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:15.193707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:15.193708 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:15.193918 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:15.194378 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:15.194385 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:15.194822 (+ 437us) client_negotiation.cc:769] Sending connection context
0810 03:53:15.194998 (+ 176us) client_negotiation.cc:240] Negotiation successful
0810 03:53:15.195332 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":116,"threads_started":1}
I20260810 03:53:15.197649 21154 heartbeater.cc:347] Connected to a master server at 127.27.153.126:45951
I20260810 03:53:15.197933 21154 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:15.198504 21154 heartbeater.cc:511] Master 127.27.153.126:45951 requested a full tablet report, sending...
I20260810 03:53:15.199757 20546 ts_manager.cc:194] Registered new tserver with Master: d020d94eb8224169bc14c10f564817f0 (127.27.153.66:39071)
I20260810 03:53:15.200363 20546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:51461
WARNING: no policy specified for kudu/127.27.153.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
I20260810 03:53:15.220952 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:45951
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:53:15.337455 21163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:15.337775 21163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:15.337827 21163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:15.341849 21163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:15.341912 21163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:15.341970 21163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:53:15.347235 21163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:45951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:15.348433 21163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:15.349455 21163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:53:15.359608 21163 server_base.cc:1034] running on GCE node
W20260810 03:53:15.359578 21171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:15.359568 21174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:15.359561 21177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:15.360342 21163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:15.360998 21163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:15.362160 21163 hybrid_clock.cc:648] HybridClock initialized: now 1786333995362139 us; error 33 us; skew 500 ppm
Aug 10 03:53:15 dist-test-slave-pww0 krb5kdc[20456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333995, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:15.364938 21163 init.cc:377] Logged in from keytab as kudu/127.27.153.67@KRBTEST.COM (short username kudu)
I20260810 03:53:15.365887 21163 webserver.cc:460] Webserver started at http://127.27.153.67:41215/ using document root <none> and password file <none>
I20260810 03:53:15.366446 21163 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:15.366519 21163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:15.366731 21163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:53:15.368894 21163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "8652e201d0ae4f138f7d7521871ac281"
format_stamp: "Formatted at 2026-08-10 03:53:15 on dist-test-slave-pww0"
server_key: "5c1124ab0fa24dc45b41c83bdac19eb9"
server_key_iv: "b3a6cb03c22adb7684d5a6cf12761caa"
server_key_version: "encryptionkey@0"
I20260810 03:53:15.369463 21163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "8652e201d0ae4f138f7d7521871ac281"
format_stamp: "Formatted at 2026-08-10 03:53:15 on dist-test-slave-pww0"
server_key: "5c1124ab0fa24dc45b41c83bdac19eb9"
server_key_iv: "b3a6cb03c22adb7684d5a6cf12761caa"
server_key_version: "encryptionkey@0"
I20260810 03:53:15.373445 21163 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:53:15.375927 21188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:15.377161 21163 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:53:15.377288 21163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "8652e201d0ae4f138f7d7521871ac281"
format_stamp: "Formatted at 2026-08-10 03:53:15 on dist-test-slave-pww0"
server_key: "5c1124ab0fa24dc45b41c83bdac19eb9"
server_key_iv: "b3a6cb03c22adb7684d5a6cf12761caa"
server_key_version: "encryptionkey@0"
I20260810 03:53:15.377393 21163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:15.388607 21163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:15.391646 21163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:15.391845 21163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:15.392464 21163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:53:15.393330 21163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:53:15.393378 21163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:15.393435 21163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:53:15.393458 21163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:15.402144 21163 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:37047
I20260810 03:53:15.402209 21388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:37047 every 8 connection(s)
I20260810 03:53:15.403123 21163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:53:15.408998 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 21163
I20260810 03:53:15.409127 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:53:15.409412 28261 external_mini_cluster.cc:1428] Setting key 763b0e81258867ee716be211f0ebb493
Aug 10 03:53:15 dist-test-slave-pww0 krb5kdc[20456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333995, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:53:15.417438 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:15.404945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:44983 (local address 127.27.153.126:45951)
0810 03:53:15.405078 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:15.405082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:15.406004 (+ 922us) server_negotiation.cc:407] Connection header received
0810 03:53:15.407516 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:15.407519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:15.407600 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:15.407684 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:15.408871 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.409627 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.411140 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.411365 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.411489 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:15.413903 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:15.413907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:15.413908 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:15.413936 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:15.415237 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:15.415834 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:15.415838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:15.415840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:15.415944 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:15.416377 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:15.416380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:15.416397 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:15.416637 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:15.416715 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:15.417204 (+ 489us) server_negotiation.cc:299] Negotiation successful
0810 03:53:15.417311 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 03:53:15.418365 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:15.405308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.67:44983)
0810 03:53:15.405829 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:15.405882 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:15.407279 (+ 1397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:15.407863 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:15.407870 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:15.408236 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:15.408726 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:15.408740 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.409790 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.409793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:15.411030 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:15.411041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.411544 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.411549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:15.411742 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:15.412274 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:15.412291 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:15.413725 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:15.415344 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:15.415350 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:15.415363 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:15.415731 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:15.416061 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:15.416064 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:15.416066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:15.416284 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:15.416734 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:15.416741 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:15.417005 (+ 264us) client_negotiation.cc:769] Sending connection context
0810 03:53:15.417172 (+ 167us) client_negotiation.cc:240] Negotiation successful
0810 03:53:15.417464 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":139,"threads_started":1}
I20260810 03:53:15.419633 21390 heartbeater.cc:347] Connected to a master server at 127.27.153.126:45951
I20260810 03:53:15.419876 21390 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:15.420383 21390 heartbeater.cc:511] Master 127.27.153.126:45951 requested a full tablet report, sending...
I20260810 03:53:15.421447 20546 ts_manager.cc:194] Registered new tserver with Master: 8652e201d0ae4f138f7d7521871ac281 (127.27.153.67:37047)
I20260810 03:53:15.422029 20546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:44983
I20260810 03:53:15.423688 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:53:15.432350 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:15.425680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55474 (local address 127.27.153.126:45951)
0810 03:53:15.425808 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:15.425812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:15.425964 (+ 152us) server_negotiation.cc:407] Connection header received
0810 03:53:15.426082 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:15.426084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:15.426123 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:15.426178 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:15.426805 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.427309 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.428162 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.428377 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.428451 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:15.429693 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:15.429696 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:15.429698 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:15.429723 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:15.430958 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:15.431327 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:15.431329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:15.431330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:15.431397 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:15.431642 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:15.431643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:15.431644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:15.431859 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:15.431962 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:15.432129 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 03:53:15.432222 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 03:53:15.435312 20546 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:53:15.438455 20546 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:53:15.453354 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:15.447603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:39071 (local address 127.0.0.1:52734)
0810 03:53:15.448472 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:15.448484 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:15.448565 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:15.449150 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:15.449153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:15.449170 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:53:15.449256 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:15.449273 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.450810 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.450812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:15.451989 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:15.451999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.452810 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.452814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:15.452920 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:15.452959 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 03:53:15.453060 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 03:53:15.453190 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":64,"thread_start_us":110,"threads_started":1}
I20260810 03:53:15.453568 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:15.447315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35158 (local address 127.27.153.67:37047)
0810 03:53:15.448395 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:15.448408 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:15.448433 (+ 25us) server_negotiation.cc:407] Connection header received
0810 03:53:15.448511 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:15.448517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:15.448725 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:15.449239 (+ 514us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:53:15.449440 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.450627 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.452155 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.452972 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.453082 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:15.453224 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:15.453324 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 03:53:15.453408 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":968,"thread_start_us":104,"threads_started":1}
I20260810 03:53:15.454036 21416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:15.448050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52734 (local address 127.27.153.66:39071)
0810 03:53:15.448597 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:15.448613 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:15.448632 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:53:15.448696 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:15.448700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:15.448860 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:15.448997 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:53:15.449420 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.450629 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.452104 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.452682 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.452782 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:15.452945 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:15.453794 (+ 849us) server_negotiation.cc:299] Negotiation successful
0810 03:53:15.453900 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":109,"threads_started":1}
I20260810 03:53:15.454051 21415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:15.448042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34134 (local address 127.27.153.65:38881)
0810 03:53:15.448860 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:15.448876 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:15.448898 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:53:15.448974 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:15.448979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:15.449141 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:15.449271 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:53:15.449522 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.450311 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.452141 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.452972 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.453082 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:15.453224 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:15.453830 (+ 606us) server_negotiation.cc:299] Negotiation successful
0810 03:53:15.453923 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":73,"threads_started":1}
I20260810 03:53:15.453559 21413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:15.447182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:37047 (local address 127.0.0.1:35158)
0810 03:53:15.447865 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:15.447915 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:15.448045 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:15.448913 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:15.448919 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:15.448943 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:53:15.449090 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:15.449098 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.450765 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.450768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:15.452032 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:15.452040 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.453099 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.453103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:15.453185 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:15.453206 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 03:53:15.453285 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 03:53:15.453346 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":86,"thread_start_us":109,"threads_started":1}
I20260810 03:53:15.453739 21411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:15.447318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:38881 (local address 127.0.0.1:34134)
0810 03:53:15.448073 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:15.448086 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:15.448180 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:15.449267 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:15.449270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:15.449296 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:53:15.449394 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:15.449401 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.450473 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.450476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:15.451989 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:15.452001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.453399 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.453402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:15.453479 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:15.453503 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:53:15.453566 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 03:53:15.453623 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":54,"thread_start_us":99,"threads_started":1}
I20260810 03:53:15.457181 20777 tablet_service.cc:1467] Processing CreateTablet for tablet d503776f93154b1e837ffb97af6555de (DEFAULT_TABLE table=test-table [id=956617ffe17146e3b0396c4a8a2679a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:53:15.457183 21267 tablet_service.cc:1467] Processing CreateTablet for tablet d503776f93154b1e837ffb97af6555de (DEFAULT_TABLE table=test-table [id=956617ffe17146e3b0396c4a8a2679a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:53:15.458223 21267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d503776f93154b1e837ffb97af6555de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:53:15.458268 20777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d503776f93154b1e837ffb97af6555de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:53:15.459925 21028 tablet_service.cc:1467] Processing CreateTablet for tablet d503776f93154b1e837ffb97af6555de (DEFAULT_TABLE table=test-table [id=956617ffe17146e3b0396c4a8a2679a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:53:15.461054 21028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d503776f93154b1e837ffb97af6555de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:53:15.465013 21417 tablet_bootstrap.cc:492] T d503776f93154b1e837ffb97af6555de P 8652e201d0ae4f138f7d7521871ac281: Bootstrap starting.
I20260810 03:53:15.466837 21418 tablet_bootstrap.cc:492] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f: Bootstrap starting.
I20260810 03:53:15.467546 21417 tablet_bootstrap.cc:654] T d503776f93154b1e837ffb97af6555de P 8652e201d0ae4f138f7d7521871ac281: Neither blocks nor log segments found. Creating new log.
I20260810 03:53:15.467741 21419 tablet_bootstrap.cc:492] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0: Bootstrap starting.
I20260810 03:53:15.469247 21418 tablet_bootstrap.cc:654] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f: Neither blocks nor log segments found. Creating new log.
I20260810 03:53:15.469710 21417 log.cc:824] T d503776f93154b1e837ffb97af6555de P 8652e201d0ae4f138f7d7521871ac281: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:15.470088 21419 tablet_bootstrap.cc:654] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0: Neither blocks nor log segments found. Creating new log.
I20260810 03:53:15.470121 21418 log.cc:824] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:15.470916 21419 log.cc:824] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:15.472154 21417 tablet_bootstrap.cc:492] T d503776f93154b1e837ffb97af6555de P 8652e201d0ae4f138f7d7521871ac281: No bootstrap required, opened a new log
I20260810 03:53:15.472190 21418 tablet_bootstrap.cc:492] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f: No bootstrap required, opened a new log
I20260810 03:53:15.472383 21417 ts_tablet_manager.cc:1397] T d503776f93154b1e837ffb97af6555de P 8652e201d0ae4f138f7d7521871ac281: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260810 03:53:15.472383 21418 ts_tablet_manager.cc:1397] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:53:15.472864 21419 tablet_bootstrap.cc:492] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0: No bootstrap required, opened a new log
I20260810 03:53:15.473045 21419 ts_tablet_manager.cc:1397] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260810 03:53:15.475697 21417 raft_consensus.cc:348] T d503776f93154b1e837ffb97af6555de P 8652e201d0ae4f138f7d7521871ac281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } }
I20260810 03:53:15.475963 21417 raft_consensus.cc:374] T d503776f93154b1e837ffb97af6555de P 8652e201d0ae4f138f7d7521871ac281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:53:15.476018 21417 raft_consensus.cc:729] T d503776f93154b1e837ffb97af6555de P 8652e201d0ae4f138f7d7521871ac281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8652e201d0ae4f138f7d7521871ac281, State: Initialized, Role: FOLLOWER
I20260810 03:53:15.476593 21417 consensus_queue.cc:260] T d503776f93154b1e837ffb97af6555de P 8652e201d0ae4f138f7d7521871ac281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } }
I20260810 03:53:15.476934 21419 raft_consensus.cc:348] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } }
I20260810 03:53:15.477191 21419 raft_consensus.cc:374] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:53:15.477248 21419 raft_consensus.cc:729] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d020d94eb8224169bc14c10f564817f0, State: Initialized, Role: FOLLOWER
I20260810 03:53:15.476934 21418 raft_consensus.cc:348] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } }
I20260810 03:53:15.477459 21418 raft_consensus.cc:374] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:53:15.477519 21418 raft_consensus.cc:729] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f28d5e87eeca486f9ab671c30f7e136f, State: Initialized, Role: FOLLOWER
I20260810 03:53:15.477775 21417 ts_tablet_manager.cc:1428] T d503776f93154b1e837ffb97af6555de P 8652e201d0ae4f138f7d7521871ac281: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 03:53:15.477891 21418 consensus_queue.cc:260] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } }
I20260810 03:53:15.478057 21390 heartbeater.cc:503] Master 127.27.153.126:45951 was elected leader, sending a full tablet report...
I20260810 03:53:15.478528 20892 heartbeater.cc:503] Master 127.27.153.126:45951 was elected leader, sending a full tablet report...
I20260810 03:53:15.478899 21418 ts_tablet_manager.cc:1428] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 03:53:15.477810 21419 consensus_queue.cc:260] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } }
I20260810 03:53:15.479859 21154 heartbeater.cc:503] Master 127.27.153.126:45951 was elected leader, sending a full tablet report...
I20260810 03:53:15.480095 21419 ts_tablet_manager.cc:1428] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260810 03:53:15.629247 21425 raft_consensus.cc:482] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:53:15.629503 21425 raft_consensus.cc:504] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } }
I20260810 03:53:15.630754 21425 leader_election.cc:288] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8652e201d0ae4f138f7d7521871ac281 (127.27.153.67:37047), f28d5e87eeca486f9ab671c30f7e136f (127.27.153.65:38881)
I20260810 03:53:15.635293 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:15.631219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:60979 (local address 127.27.153.67:37047)
0810 03:53:15.631379 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:15.631383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:15.631402 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:53:15.631493 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:15.631496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:15.631551 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:15.631618 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:53:15.631936 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.632651 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.634121 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.634742 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.634869 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:15.634927 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:15.635080 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 03:53:15.635158 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 03:53:15.635308 21157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:15.631068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:37047 (local address 127.27.153.66:60979)
0810 03:53:15.631204 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:15.631221 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:15.631356 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:15.631654 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:15.631658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:15.631679 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:53:15.631830 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:15.631837 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.632747 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.632750 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:15.633990 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:15.634001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.634864 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.634867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:15.634957 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:15.634984 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 03:53:15.635061 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 03:53:15.635140 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"spinlock_wait_cycles":1280}
I20260810 03:53:15.635488 21415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:15.631187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:47083 (local address 127.27.153.65:38881)
0810 03:53:15.631305 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:15.631310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:15.631749 (+ 439us) server_negotiation.cc:407] Connection header received
0810 03:53:15.631861 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:15.631864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:15.631909 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:15.632012 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:53:15.632287 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.632928 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.634291 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.634907 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.635018 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:15.635065 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:15.635265 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 03:53:15.635354 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:53:15.635923 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:15.631068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:38881 (local address 127.27.153.66:47083)
0810 03:53:15.631620 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:15.631633 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:15.631762 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:15.632005 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:15.632007 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:15.632070 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:53:15.632183 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:15.632189 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.633024 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.633027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:15.634178 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:15.634185 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.635024 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.635028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:15.635121 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:15.635141 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 03:53:15.635672 (+ 531us) client_negotiation.cc:240] Negotiation successful
0810 03:53:15.635753 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":125,"spinlock_wait_cycles":30848,"thread_start_us":175,"threads_started":1}
I20260810 03:53:15.636085 21296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d503776f93154b1e837ffb97af6555de" candidate_uuid: "d020d94eb8224169bc14c10f564817f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8652e201d0ae4f138f7d7521871ac281" is_pre_election: true
I20260810 03:53:15.636483 21296 raft_consensus.cc:2454] T d503776f93154b1e837ffb97af6555de P 8652e201d0ae4f138f7d7521871ac281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d020d94eb8224169bc14c10f564817f0 in term 0.
I20260810 03:53:15.636647 20806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d503776f93154b1e837ffb97af6555de" candidate_uuid: "d020d94eb8224169bc14c10f564817f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f28d5e87eeca486f9ab671c30f7e136f" is_pre_election: true
I20260810 03:53:15.636940 20806 raft_consensus.cc:2454] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d020d94eb8224169bc14c10f564817f0 in term 0.
I20260810 03:53:15.636966 20940 leader_election.cc:302] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8652e201d0ae4f138f7d7521871ac281, d020d94eb8224169bc14c10f564817f0; no voters:
I20260810 03:53:15.637235 21425 raft_consensus.cc:2781] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:53:15.637346 21425 raft_consensus.cc:482] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:53:15.637399 21425 raft_consensus.cc:3037] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:53:15.638473 21425 raft_consensus.cc:504] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } }
I20260810 03:53:15.638801 21425 leader_election.cc:288] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0 [CANDIDATE]: Term 1 election: Requested vote from peers 8652e201d0ae4f138f7d7521871ac281 (127.27.153.67:37047), f28d5e87eeca486f9ab671c30f7e136f (127.27.153.65:38881)
I20260810 03:53:15.639209 21296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d503776f93154b1e837ffb97af6555de" candidate_uuid: "d020d94eb8224169bc14c10f564817f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8652e201d0ae4f138f7d7521871ac281"
I20260810 03:53:15.639213 20806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d503776f93154b1e837ffb97af6555de" candidate_uuid: "d020d94eb8224169bc14c10f564817f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f28d5e87eeca486f9ab671c30f7e136f"
I20260810 03:53:15.639345 21296 raft_consensus.cc:3037] T d503776f93154b1e837ffb97af6555de P 8652e201d0ae4f138f7d7521871ac281 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:53:15.639335 20806 raft_consensus.cc:3037] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:53:15.640414 20806 raft_consensus.cc:2454] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d020d94eb8224169bc14c10f564817f0 in term 1.
I20260810 03:53:15.640425 21296 raft_consensus.cc:2454] T d503776f93154b1e837ffb97af6555de P 8652e201d0ae4f138f7d7521871ac281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d020d94eb8224169bc14c10f564817f0 in term 1.
I20260810 03:53:15.640704 20940 leader_election.cc:302] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8652e201d0ae4f138f7d7521871ac281, d020d94eb8224169bc14c10f564817f0, f28d5e87eeca486f9ab671c30f7e136f; no voters:
I20260810 03:53:15.640893 21425 raft_consensus.cc:2781] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:53:15.641129 21425 raft_consensus.cc:686] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0 [term 1 LEADER]: Becoming Leader. State: Replica: d020d94eb8224169bc14c10f564817f0, State: Running, Role: LEADER
I20260810 03:53:15.641499 21425 consensus_queue.cc:237] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } }
I20260810 03:53:15.644768 20546 catalog_manager.cc:5495] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to d020d94eb8224169bc14c10f564817f0 (127.27.153.66). New cstate: current_term: 1 leader_uuid: "d020d94eb8224169bc14c10f564817f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } health_report { overall_health: HEALTHY } } }
W20260810 03:53:15.654238 21393 tablet.cc:2369] T d503776f93154b1e837ffb97af6555de P 8652e201d0ae4f138f7d7521871ac281: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:53:15.681326 21155 tablet.cc:2369] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 03:53:15.685125 20895 tablet.cc:2369] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:53:15.690956 21416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:15.687230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52744 (local address 127.27.153.66:39071)
0810 03:53:15.687371 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:15.687376 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:15.687391 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:53:15.687561 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:15.687564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:15.687625 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:15.687727 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:53:15.687985 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.688502 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.689403 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:15.689619 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:15.689708 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:15.690037 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:53:15.690544 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:53:15.690664 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:15.690741 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 03:53:15.690808 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:53:15.697146 20806 raft_consensus.cc:1261] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Refusing update from remote peer d020d94eb8224169bc14c10f564817f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:53:15.697154 21296 raft_consensus.cc:1261] T d503776f93154b1e837ffb97af6555de P 8652e201d0ae4f138f7d7521871ac281 [term 1 FOLLOWER]: Refusing update from remote peer d020d94eb8224169bc14c10f564817f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:53:15.698007 21427 consensus_queue.cc:1035] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:53:15.698232 21425 consensus_queue.cc:1035] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:53:15.706212 21430 mvcc.cc:205] Tried to move back new op lower bound from 7316824046367703040 to 7316824046150258688. Current Snapshot: MvccSnapshot[applied={T|T < 7316824046367703040}]
I20260810 03:53:15.708092 21431 mvcc.cc:205] Tried to move back new op lower bound from 7316824046367703040 to 7316824046150258688. Current Snapshot: MvccSnapshot[applied={T|T < 7316824046367703040}]
I20260810 03:53:15.708465 21432 mvcc.cc:205] Tried to move back new op lower bound from 7316824046367703040 to 7316824046150258688. Current Snapshot: MvccSnapshot[applied={T|T < 7316824046367703040}]
I20260810 03:53:15.716270 20546 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:53:15.716452 20546 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:53:15.719053 20546 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c: Sending DeleteTablet for 3 replicas of tablet d503776f93154b1e837ffb97af6555de
I20260810 03:53:15.719995 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 20657
I20260810 03:53:15.719964 21267 tablet_service.cc:1514] Processing DeleteTablet for tablet d503776f93154b1e837ffb97af6555de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:53:15 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:35158
I20260810 03:53:15.719967 20777 tablet_service.cc:1514] Processing DeleteTablet for tablet d503776f93154b1e837ffb97af6555de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:53:15 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:34134
I20260810 03:53:15.720240 21028 tablet_service.cc:1514] Processing DeleteTablet for tablet d503776f93154b1e837ffb97af6555de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:53:15 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:52734
I20260810 03:53:15.747908 21443 tablet_replica.cc:330] T d503776f93154b1e837ffb97af6555de P 8652e201d0ae4f138f7d7521871ac281: stopping tablet replica
I20260810 03:53:15.747931 21444 tablet_replica.cc:330] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0: stopping tablet replica
I20260810 03:53:15.748377 21443 raft_consensus.cc:2229] T d503776f93154b1e837ffb97af6555de P 8652e201d0ae4f138f7d7521871ac281 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:53:15.748416 21444 raft_consensus.cc:2229] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:53:15.748781 21443 raft_consensus.cc:2258] T d503776f93154b1e837ffb97af6555de P 8652e201d0ae4f138f7d7521871ac281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:53:15.748857 21444 raft_consensus.cc:2258] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:53:15.750504 21444 ts_tablet_manager.cc:1905] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:53:15.750541 21443 ts_tablet_manager.cc:1905] T d503776f93154b1e837ffb97af6555de P 8652e201d0ae4f138f7d7521871ac281: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:53:15.753414 21443 ts_tablet_manager.cc:1918] T d503776f93154b1e837ffb97af6555de P 8652e201d0ae4f138f7d7521871ac281: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:53:15.753537 21443 log.cc:1196] T d503776f93154b1e837ffb97af6555de P 8652e201d0ae4f138f7d7521871ac281: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/d503776f93154b1e837ffb97af6555de
I20260810 03:53:15.753520 21444 ts_tablet_manager.cc:1918] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:53:15.753635 21444 log.cc:1196] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/d503776f93154b1e837ffb97af6555de
I20260810 03:53:15.753875 21443 ts_tablet_manager.cc:1939] T d503776f93154b1e837ffb97af6555de P 8652e201d0ae4f138f7d7521871ac281: Deleting consensus metadata
I20260810 03:53:15.753959 21444 ts_tablet_manager.cc:1939] T d503776f93154b1e837ffb97af6555de P d020d94eb8224169bc14c10f564817f0: Deleting consensus metadata
I20260810 03:53:15.755308 20509 catalog_manager.cc:4841] TS d020d94eb8224169bc14c10f564817f0 (127.27.153.66:39071): tablet d503776f93154b1e837ffb97af6555de (table test-table [id=956617ffe17146e3b0396c4a8a2679a2]) successfully deleted
I20260810 03:53:15.755620 20510 catalog_manager.cc:4841] TS 8652e201d0ae4f138f7d7521871ac281 (127.27.153.67:37047): tablet d503776f93154b1e837ffb97af6555de (table test-table [id=956617ffe17146e3b0396c4a8a2679a2]) successfully deleted
W20260810 03:53:15.756299 20513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.153.65:38881 (error 108)
I20260810 03:53:15.756549 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 20906
I20260810 03:53:15.778642 21411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:15.757178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:38881 (local address 127.0.0.1:34140)
0810 03:53:15.778414 (+ 21236us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:15.778539 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:38881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21071}
W20260810 03:53:15.778936 20513 catalog_manager.cc:4584] TS f28d5e87eeca486f9ab671c30f7e136f (127.27.153.65:38881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d503776f93154b1e837ffb97af6555de: Network error: Client connection negotiation failed: client connection to 127.27.153.65:38881: connect: Connection refused (error 111)
I20260810 03:53:15.784051 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 21163
I20260810 03:53:15.801016 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 20480
I20260810 03:53:15.825148 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:45951
--webserver_interface=127.27.153.126
--webserver_port=34513
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:45951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:53:15.945030 21452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:15.945334 21452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:15.945377 21452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:15.949555 21452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:53:15.949632 21452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:15.949651 21452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:15.949666 21452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:53:15.949679 21452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:53:15.955251 21452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:45951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:45951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=34513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:15.956463 21452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:15.957562 21452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:15.972551 21462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:15.972728 21452 server_base.cc:1034] running on GCE node
W20260810 03:53:15.972544 21463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:15.972994 21468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:15.973701 21452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:15.974993 21452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:15.976188 21452 hybrid_clock.cc:648] HybridClock initialized: now 1786333995976146 us; error 53 us; skew 500 ppm
Aug 10 03:53:15 dist-test-slave-pww0 krb5kdc[20456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333995, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:15.979573 21452 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:53:15.980613 21452 webserver.cc:460] Webserver started at http://127.27.153.126:34513/ using document root <none> and password file <none>
I20260810 03:53:15.981216 21452 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:15.981300 21452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:15.985740 21452 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:53:15.988186 21480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:15.989598 21452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:53:15.989760 21452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "6709cbea73ae4017848debfebb08f24c"
format_stamp: "Formatted at 2026-08-10 03:53:14 on dist-test-slave-pww0"
server_key: "9f89fede1da93d8154db3a2b3afd1c31"
server_key_iv: "0d1fabdae75bbd2557a31505da525274"
server_key_version: "encryptionkey@0"
I20260810 03:53:15.990483 21452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:16.002926 21452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:16.006410 21452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:16.006615 21452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:16.014225 21452 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:45951
I20260810 03:53:16.014586 21587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:45951 every 8 connection(s)
I20260810 03:53:16.015496 21452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:53:16.020488 21591 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:53:16.022337 21591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c: Bootstrap starting.
I20260810 03:53:16.023068 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 21452
I20260810 03:53:16.023582 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:38881
--local_ip_for_outbound_sockets=127.27.153.65
--tserver_master_addrs=127.27.153.126:45951
--webserver_port=38537
--webserver_interface=127.27.153.65
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:53:16.041445 21591 log.cc:824] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:16.051021 21591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:53:16.051434 21591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c: Bootstrap complete.
I20260810 03:53:16.055717 21591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709cbea73ae4017848debfebb08f24c" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45951 } }
I20260810 03:53:16.056176 21591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6709cbea73ae4017848debfebb08f24c, State: Initialized, Role: FOLLOWER
I20260810 03:53:16.056697 21591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709cbea73ae4017848debfebb08f24c" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45951 } }
I20260810 03:53:16.056826 21591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:53:16.056898 21591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:53:16.056996 21591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:53:16.059166 21591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709cbea73ae4017848debfebb08f24c" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45951 } }
I20260810 03:53:16.059590 21591 leader_election.cc:302] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6709cbea73ae4017848debfebb08f24c; no voters:
I20260810 03:53:16.059928 21591 leader_election.cc:288] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 03:53:16.060037 21599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 2 FOLLOWER]: Leader election won for term 2
I20260810 03:53:16.060240 21599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 2 LEADER]: Becoming Leader. State: Replica: 6709cbea73ae4017848debfebb08f24c, State: Running, Role: LEADER
I20260810 03:53:16.060640 21599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709cbea73ae4017848debfebb08f24c" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45951 } }
I20260810 03:53:16.061291 21591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:53:16.062158 21604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6709cbea73ae4017848debfebb08f24c. Latest consensus state: current_term: 2 leader_uuid: "6709cbea73ae4017848debfebb08f24c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709cbea73ae4017848debfebb08f24c" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45951 } } }
I20260810 03:53:16.062295 21604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [sys.catalog]: This master's current role is: LEADER
I20260810 03:53:16.062654 21614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:53:16.062671 21600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6709cbea73ae4017848debfebb08f24c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709cbea73ae4017848debfebb08f24c" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45951 } } }
I20260810 03:53:16.062784 21600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [sys.catalog]: This master's current role is: LEADER
I20260810 03:53:16.067622 21614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:53:16.068315 21614 catalog_manager.cc:1259] Loaded cluster ID: 20365940fafd4bd18913a6405bb087c3
I20260810 03:53:16.068373 21614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:53:16.069603 21614 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 03:53:16.072463 21614 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 03:53:16.077430 21614 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:53:16.078305 21614 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c: Loaded TSK: 0
I20260810 03:53:16.079339 21614 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 03:53:16.168267 21596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:16.168642 21596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:16.168705 21596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:16.173017 21596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:16.173094 21596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:16.173156 21596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:53:16.178751 21596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:38881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=38537
--tserver_master_addrs=127.27.153.126:45951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:16.179957 21596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:16.181053 21596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:16.191184 21641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:16.191185 21640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:16.191344 21645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:16.191959 21596 server_base.cc:1034] running on GCE node
I20260810 03:53:16.192391 21596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:16.193100 21596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:16.194270 21596 hybrid_clock.cc:648] HybridClock initialized: now 1786333996194234 us; error 53 us; skew 500 ppm
Aug 10 03:53:16 dist-test-slave-pww0 krb5kdc[20456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333996, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:16.197356 21596 init.cc:377] Logged in from keytab as kudu/127.27.153.65@KRBTEST.COM (short username kudu)
I20260810 03:53:16.198378 21596 webserver.cc:460] Webserver started at http://127.27.153.65:38537/ using document root <none> and password file <none>
I20260810 03:53:16.198961 21596 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:16.199057 21596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:16.203186 21596 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:53:16.205598 21661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:16.206948 21596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 03:53:16.207075 21596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "f28d5e87eeca486f9ab671c30f7e136f"
format_stamp: "Formatted at 2026-08-10 03:53:14 on dist-test-slave-pww0"
server_key: "6c9c1f5ba883dad47917d6f5eaddc93d"
server_key_iv: "9c78c69df8871b6c9c0ea1bfc112b776"
server_key_version: "encryptionkey@0"
I20260810 03:53:16.207708 21596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:16.223412 21596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:16.227566 21596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:16.227806 21596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:16.228488 21596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:53:16.229789 21675 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 03:53:16.235276 21596 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 03:53:16.235373 21596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260810 03:53:16.235481 21596 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 03:53:16.237331 21596 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 03:53:16.237385 21596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 03:53:16.237711 21675 tablet_bootstrap.cc:492] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f: Bootstrap starting.
I20260810 03:53:16.247145 21596 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:38881
I20260810 03:53:16.247232 21868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:38881 every 8 connection(s)
I20260810 03:53:16.248181 21596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:53:16.252671 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 21596
I20260810 03:53:16.253157 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:39071
--local_ip_for_outbound_sockets=127.27.153.66
--tserver_master_addrs=127.27.153.126:45951
--webserver_port=38735
--webserver_interface=127.27.153.66
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:53:16.257546 21675 log.cc:824] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:53:16 dist-test-slave-pww0 krb5kdc[20456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333996, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:53:16.266556 21675 tablet_bootstrap.cc:492] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:53:16.266937 21675 tablet_bootstrap.cc:492] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f: Bootstrap complete.
I20260810 03:53:16.268155 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:16.250145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.65:54579)
0810 03:53:16.250763 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:16.250840 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:16.252002 (+ 1162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:16.253515 (+ 1513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:16.253524 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:16.253910 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:16.254402 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:16.254418 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.256140 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.256144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:16.257515 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:16.257527 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.258539 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.258543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:16.258750 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:16.259503 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:16.259525 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:16.261562 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:16.264445 (+ 2883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:16.264452 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:16.264468 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:16.264838 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:16.265264 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:16.265267 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:16.265269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:16.265461 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:16.266207 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:16.266214 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:16.266542 (+ 328us) client_negotiation.cc:769] Sending connection context
0810 03:53:16.266753 (+ 211us) client_negotiation.cc:240] Negotiation successful
0810 03:53:16.267126 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":151,"threads_started":1}
I20260810 03:53:16.268210 21675 ts_tablet_manager.cc:1397] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f: Time spent bootstrapping tablet: real 0.031s user 0.028s sys 0.002s
I20260810 03:53:16.269241 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:16.250244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:54579 (local address 127.27.153.126:45951)
0810 03:53:16.250771 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:16.250790 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:16.250973 (+ 183us) server_negotiation.cc:407] Connection header received
0810 03:53:16.252502 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:16.252513 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:16.252881 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:16.253220 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:16.254593 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.255968 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.257670 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.258368 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.259198 (+ 830us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:16.261796 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:16.261803 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:16.261822 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:16.261859 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:16.264298 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:16.264983 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:16.264990 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:16.264995 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:16.265160 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:16.265566 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:16.265569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:16.265577 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:16.266076 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:16.266220 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:16.267682 (+ 1462us) server_negotiation.cc:299] Negotiation successful
0810 03:53:16.268008 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":155,"threads_started":1}
I20260810 03:53:16.271126 21675 raft_consensus.cc:348] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } }
I20260810 03:53:16.271680 21675 raft_consensus.cc:729] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f28d5e87eeca486f9ab671c30f7e136f, State: Initialized, Role: FOLLOWER
I20260810 03:53:16.271822 21869 heartbeater.cc:347] Connected to a master server at 127.27.153.126:45951
I20260810 03:53:16.272073 21869 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:16.272168 21675 consensus_queue.cc:260] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } }
I20260810 03:53:16.272663 21869 heartbeater.cc:511] Master 127.27.153.126:45951 requested a full tablet report, sending...
I20260810 03:53:16.273187 21675 ts_tablet_manager.cc:1428] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260810 03:53:16.275686 21519 ts_manager.cc:194] Registered new tserver with Master: f28d5e87eeca486f9ab671c30f7e136f (127.27.153.65:38881)
I20260810 03:53:16.280051 21519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:54579
I20260810 03:53:16.282820 21869 heartbeater.cc:503] Master 127.27.153.126:45951 was elected leader, sending a full tablet report...
Aug 10 03:53:16 dist-test-slave-pww0 krb5kdc[20456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333995, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for kudu/127.27.153.65@KRBTEST.COM
I20260810 03:53:16.289785 21879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:16.279305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:38881 (local address 127.0.0.1:34154)
0810 03:53:16.280093 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:16.280130 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:16.280281 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:16.280787 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:16.280793 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:16.281243 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:16.281365 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:16.281372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.282309 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.282312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:16.282935 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:16.282946 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.283722 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.283726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:16.283852 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:16.284586 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:16.284600 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:16.286181 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:16.288008 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:16.288018 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:16.288024 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:16.288269 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:16.288561 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:16.288564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:16.288566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:16.288653 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:16.289114 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:16.289120 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:16.289245 (+ 125us) client_negotiation.cc:769] Sending connection context
0810 03:53:16.289372 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 03:53:16.289593 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":157,"threads_started":1}
I20260810 03:53:16.289789 21880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:16.279377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34154 (local address 127.27.153.65:38881)
0810 03:53:16.279843 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:16.279855 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:16.280401 (+ 546us) server_negotiation.cc:407] Connection header received
0810 03:53:16.280469 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:16.280473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:16.280636 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:16.280778 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:16.281481 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.282193 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.283058 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.283494 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.283591 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:16.286325 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:16.286335 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:16.286346 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:16.286370 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:16.287900 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:16.288382 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:16.288386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:16.288388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:16.288471 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:16.288735 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:16.288738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:16.288740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:16.289002 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:16.289122 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:16.289468 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 03:53:16.289634 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":115,"threads_started":1}
I20260810 03:53:16.290836 21752 tablet_service.cc:1514] Processing DeleteTablet for tablet d503776f93154b1e837ffb97af6555de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:53:15 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:34154
I20260810 03:53:16.291767 21881 tablet_replica.cc:330] stopping tablet replica
I20260810 03:53:16.292124 21881 raft_consensus.cc:2229] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:53:16.292412 21881 raft_consensus.cc:2258] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:53:16.293591 21881 ts_tablet_manager.cc:1905] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:53:16.298439 21881 ts_tablet_manager.cc:1918] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:53:16.298569 21881 log.cc:1196] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/wal/wals/d503776f93154b1e837ffb97af6555de
I20260810 03:53:16.298851 21881 ts_tablet_manager.cc:1939] T d503776f93154b1e837ffb97af6555de P f28d5e87eeca486f9ab671c30f7e136f: Deleting consensus metadata
I20260810 03:53:16.301100 21487 catalog_manager.cc:4841] TS f28d5e87eeca486f9ab671c30f7e136f (127.27.153.65:38881): tablet d503776f93154b1e837ffb97af6555de (table test-table [id=956617ffe17146e3b0396c4a8a2679a2]) successfully deleted
W20260810 03:53:16.383033 21874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:16.383425 21874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:16.383523 21874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:16.390204 21874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:16.390322 21874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:16.390442 21874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:53:16.396471 21874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:39071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=38735
--tserver_master_addrs=127.27.153.126:45951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:16.397640 21874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:16.398726 21874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:16.408524 21893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:16.408524 21888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:16.408589 21889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:16.408604 21874 server_base.cc:1034] running on GCE node
I20260810 03:53:16.409205 21874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:16.409891 21874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:16.411055 21874 hybrid_clock.cc:648] HybridClock initialized: now 1786333996411035 us; error 37 us; skew 500 ppm
Aug 10 03:53:16 dist-test-slave-pww0 krb5kdc[20456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333996, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:16.413973 21874 init.cc:377] Logged in from keytab as kudu/127.27.153.66@KRBTEST.COM (short username kudu)
I20260810 03:53:16.414928 21874 webserver.cc:460] Webserver started at http://127.27.153.66:38735/ using document root <none> and password file <none>
I20260810 03:53:16.415520 21874 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:16.415608 21874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:16.419632 21874 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:53:16.421828 21901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:16.423166 21874 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 03:53:16.423280 21874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "d020d94eb8224169bc14c10f564817f0"
format_stamp: "Formatted at 2026-08-10 03:53:15 on dist-test-slave-pww0"
server_key: "c720c48cd55f9e3224da960677eb8b1f"
server_key_iv: "c8853d6c6ae3ab4d2c44050b8cddf8a5"
server_key_version: "encryptionkey@0"
I20260810 03:53:16.423938 21874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:16.449597 21874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:16.453076 21874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:16.453275 21874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:16.453922 21874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:53:16.454818 21874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:53:16.454875 21874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:16.454926 21874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:53:16.454955 21874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:16.464255 21874 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:39071
I20260810 03:53:16.464285 22107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:39071 every 8 connection(s)
I20260810 03:53:16.465332 21874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:53:16.473410 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 21874
I20260810 03:53:16.473848 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:37047
--local_ip_for_outbound_sockets=127.27.153.67
--tserver_master_addrs=127.27.153.126:45951
--webserver_port=41215
--webserver_interface=127.27.153.67
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:53:16 dist-test-slave-pww0 krb5kdc[20456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333996, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:53:16.480021 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:16.467436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:41393 (local address 127.27.153.126:45951)
0810 03:53:16.467588 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:16.467591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:16.468517 (+ 926us) server_negotiation.cc:407] Connection header received
0810 03:53:16.469858 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:16.469862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:16.469924 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:16.470000 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:16.471252 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.471838 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.473023 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.473334 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.473449 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:16.476166 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:16.476169 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:16.476171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:16.476202 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:16.477771 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:16.478295 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:16.478299 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:16.478300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:16.478408 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:16.478717 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:16.478720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:16.478722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:16.478991 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:16.479102 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:16.479687 (+ 585us) server_negotiation.cc:299] Negotiation successful
0810 03:53:16.479842 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:53:16.480847 22110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:16.467842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.66:41393)
0810 03:53:16.468351 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:16.468399 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:16.469631 (+ 1232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:16.470190 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:16.470197 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:16.470584 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:16.471101 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:16.471117 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.471964 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.471967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:16.472891 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:16.472902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.473502 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.473506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:16.473695 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:16.474312 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:16.474329 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:16.475934 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:16.477904 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:16.477909 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:16.477924 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:16.478191 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:16.478491 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:16.478494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:16.478495 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:16.478636 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:16.479109 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:16.479115 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:16.479403 (+ 288us) client_negotiation.cc:769] Sending connection context
0810 03:53:16.479610 (+ 207us) client_negotiation.cc:240] Negotiation successful
0810 03:53:16.479910 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":120,"threads_started":1}
I20260810 03:53:16.482065 22108 heartbeater.cc:347] Connected to a master server at 127.27.153.126:45951
I20260810 03:53:16.482367 22108 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:16.483053 22108 heartbeater.cc:511] Master 127.27.153.126:45951 requested a full tablet report, sending...
I20260810 03:53:16.484328 21519 ts_manager.cc:194] Registered new tserver with Master: d020d94eb8224169bc14c10f564817f0 (127.27.153.66:39071)
I20260810 03:53:16.484932 21519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:41393
W20260810 03:53:16.593284 22111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:16.593583 22111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:16.593628 22111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:16.597862 22111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:16.597945 22111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:16.598012 22111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:53:16.603682 22111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:37047
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=41215
--tserver_master_addrs=127.27.153.126:45951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:16.604957 22111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:16.606034 22111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:16.615816 22131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:16.615880 22126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:16.615840 22124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:16.616056 22111 server_base.cc:1034] running on GCE node
I20260810 03:53:16.616506 22111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:16.617129 22111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:16.618292 22111 hybrid_clock.cc:648] HybridClock initialized: now 1786333996618266 us; error 41 us; skew 500 ppm
Aug 10 03:53:16 dist-test-slave-pww0 krb5kdc[20456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333996, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:16.621012 22111 init.cc:377] Logged in from keytab as kudu/127.27.153.67@KRBTEST.COM (short username kudu)
I20260810 03:53:16.621897 22111 webserver.cc:460] Webserver started at http://127.27.153.67:41215/ using document root <none> and password file <none>
I20260810 03:53:16.622452 22111 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:16.622538 22111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:16.626734 22111 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:53:16.629032 22146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:16.630330 22111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 03:53:16.630450 22111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "8652e201d0ae4f138f7d7521871ac281"
format_stamp: "Formatted at 2026-08-10 03:53:15 on dist-test-slave-pww0"
server_key: "5c1124ab0fa24dc45b41c83bdac19eb9"
server_key_iv: "b3a6cb03c22adb7684d5a6cf12761caa"
server_key_version: "encryptionkey@0"
I20260810 03:53:16.631016 22111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:16.671337 22111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:16.674741 22111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:16.674958 22111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:16.675614 22111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:53:16.676506 22111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:53:16.676558 22111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:16.676613 22111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:53:16.676640 22111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:16.685719 22111 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:37047
I20260810 03:53:16.685760 22343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:37047 every 8 connection(s)
I20260810 03:53:16.686847 22111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:53:16.693251 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 22111
Aug 10 03:53:16 dist-test-slave-pww0 krb5kdc[20456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333996, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:53:16.702059 22349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:16.693999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.27.153.126:45951)
0810 03:53:16.694306 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:16.694310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:16.694386 (+ 76us) server_negotiation.cc:407] Connection header received
0810 03:53:16.694496 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:16.694499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:16.694545 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:16.694640 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:16.695429 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.696006 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.696891 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.697127 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.697235 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:16.698607 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:16.698610 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:16.698613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:16.698656 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:16.700095 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:16.700560 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:16.700563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:16.700564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:16.700703 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:16.701036 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:16.701039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:16.701040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:16.701393 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:16.701604 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:16.701787 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 03:53:16.701905 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":234,"spinlock_wait_cycles":8320,"thread_start_us":108,"threads_started":1}
I20260810 03:53:16.703002 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:16.688834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:43009 (local address 127.27.153.126:45951)
0810 03:53:16.689001 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:16.689005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:16.689951 (+ 946us) server_negotiation.cc:407] Connection header received
0810 03:53:16.691371 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:16.691374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:16.691428 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:16.691549 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:16.692836 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.693398 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.695017 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.695329 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.695431 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:16.698150 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:16.698154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:16.698157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:16.698182 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:16.699852 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:16.700551 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:16.700555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:16.700557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:16.700695 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:16.701109 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:16.701113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:16.701114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:16.701393 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:16.701501 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:16.702716 (+ 1215us) server_negotiation.cc:299] Negotiation successful
0810 03:53:16.702848 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":70,"spinlock_wait_cycles":1920}
I20260810 03:53:16.703662 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:16.689221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.67:43009)
0810 03:53:16.689778 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:16.689828 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:16.691140 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:16.691745 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:16.691752 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:16.692147 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:16.692689 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:16.692706 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.693537 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.693541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:16.694901 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:16.694913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.695507 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.695512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:16.695716 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:16.696298 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:16.696316 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:16.697969 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:16.699979 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:16.699985 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:16.700001 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:16.700428 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:16.700788 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:16.700791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:16.700793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:16.701005 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:16.701502 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:16.701509 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:16.701875 (+ 366us) client_negotiation.cc:769] Sending connection context
0810 03:53:16.702148 (+ 273us) client_negotiation.cc:240] Negotiation successful
0810 03:53:16.702551 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":134,"threads_started":1}
I20260810 03:53:16.704821 22344 heartbeater.cc:347] Connected to a master server at 127.27.153.126:45951
I20260810 03:53:16.705104 22344 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:16.705681 22344 heartbeater.cc:511] Master 127.27.153.126:45951 requested a full tablet report, sending...
I20260810 03:53:16.706836 21519 ts_manager.cc:194] Registered new tserver with Master: 8652e201d0ae4f138f7d7521871ac281 (127.27.153.67:37047)
I20260810 03:53:16.707406 21519 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:43009
I20260810 03:53:16.716620 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:53:16.721027 21871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:16.717306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55486 (local address 127.27.153.126:45951)
0810 03:53:16.717459 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:16.717464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:16.717624 (+ 160us) server_negotiation.cc:407] Connection header received
0810 03:53:16.717810 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:16.717812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:16.717930 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:16.718032 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:53:16.718309 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.718830 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.719677 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.719904 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.719984 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:16.720232 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:53:16.720667 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:53:16.720758 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:16.720825 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 03:53:16.720879 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 03:53:16.721755 21519 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:55486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:53:16.725445 21519 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:53:16.735404 21752 tablet_service.cc:1467] Processing CreateTablet for tablet 82255aab37ba4f71b6245f39774e9cae (DEFAULT_TABLE table=test-table [id=c218746065be46ffa37cc242f4e8a649]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:53:16.736187 21752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82255aab37ba4f71b6245f39774e9cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:53:16.739168 21675 tablet_bootstrap.cc:492] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f: Bootstrap starting.
I20260810 03:53:16.739346 21879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:16.733544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:37047 (local address 127.0.0.1:35170)
0810 03:53:16.733799 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:16.733813 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:16.733892 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:16.734686 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:16.734689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:16.734710 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:53:16.734828 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:16.734836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.736636 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.736640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:16.737887 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:16.737899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.738951 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.738955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:16.739050 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:16.739075 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 03:53:16.739127 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 03:53:16.739208 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":57}
I20260810 03:53:16.739683 22362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:16.734228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52746 (local address 127.27.153.66:39071)
0810 03:53:16.734614 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:16.734634 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:16.734657 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:53:16.734725 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:16.734730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:16.734977 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:16.735138 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:53:16.735373 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.736546 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.737799 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.738616 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.738736 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:16.738904 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:16.739397 (+ 493us) server_negotiation.cc:299] Negotiation successful
0810 03:53:16.739536 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":162,"threads_started":1}
I20260810 03:53:16.739984 22359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:16.733755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35170 (local address 127.27.153.67:37047)
0810 03:53:16.734191 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:16.734203 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:16.734226 (+ 23us) server_negotiation.cc:407] Connection header received
0810 03:53:16.734296 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:16.734301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:16.734512 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:16.734699 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:53:16.734941 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.736479 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.738023 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.738816 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.739552 (+ 736us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:16.739720 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:16.739774 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 03:53:16.739873 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":169,"threads_started":1}
I20260810 03:53:16.740170 21675 tablet_bootstrap.cc:654] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f: Neither blocks nor log segments found. Creating new log.
I20260810 03:53:16.740305 22357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:16.733807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:39071 (local address 127.0.0.1:52746)
0810 03:53:16.734217 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:16.734229 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:16.734331 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:16.735119 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:16.735122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:16.735140 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:53:16.735228 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:16.735234 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.736692 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.736695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:16.737667 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:16.737678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.738772 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.738776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:16.738872 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:16.738938 (+ 66us) client_negotiation.cc:769] Sending connection context
0810 03:53:16.739024 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 03:53:16.739115 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":28,"thread_start_us":126,"threads_started":1}
I20260810 03:53:16.741555 21675 tablet_bootstrap.cc:492] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f: No bootstrap required, opened a new log
I20260810 03:53:16.741667 21675 ts_tablet_manager.cc:1397] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 03:53:16.742129 21675 raft_consensus.cc:348] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } }
I20260810 03:53:16.742246 21675 raft_consensus.cc:374] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:53:16.742282 21675 raft_consensus.cc:729] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f28d5e87eeca486f9ab671c30f7e136f, State: Initialized, Role: FOLLOWER
I20260810 03:53:16.742416 21675 consensus_queue.cc:260] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } }
I20260810 03:53:16.742683 21675 ts_tablet_manager.cc:1428] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 03:53:16.743288 22234 tablet_service.cc:1467] Processing CreateTablet for tablet 82255aab37ba4f71b6245f39774e9cae (DEFAULT_TABLE table=test-table [id=c218746065be46ffa37cc242f4e8a649]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:53:16.743649 21998 tablet_service.cc:1467] Processing CreateTablet for tablet 82255aab37ba4f71b6245f39774e9cae (DEFAULT_TABLE table=test-table [id=c218746065be46ffa37cc242f4e8a649]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:53:16.744496 21998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82255aab37ba4f71b6245f39774e9cae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:53:16.744546 22234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82255aab37ba4f71b6245f39774e9cae. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 03:53:16.749440 21870 tablet.cc:2369] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:53:16.752094 22368 tablet_bootstrap.cc:492] T 82255aab37ba4f71b6245f39774e9cae P d020d94eb8224169bc14c10f564817f0: Bootstrap starting.
I20260810 03:53:16.752094 22367 tablet_bootstrap.cc:492] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281: Bootstrap starting.
I20260810 03:53:16.754897 22368 tablet_bootstrap.cc:654] T 82255aab37ba4f71b6245f39774e9cae P d020d94eb8224169bc14c10f564817f0: Neither blocks nor log segments found. Creating new log.
I20260810 03:53:16.754918 22367 tablet_bootstrap.cc:654] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281: Neither blocks nor log segments found. Creating new log.
I20260810 03:53:16.755868 22368 log.cc:824] T 82255aab37ba4f71b6245f39774e9cae P d020d94eb8224169bc14c10f564817f0: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:16.755870 22367 log.cc:824] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:16.758067 22368 tablet_bootstrap.cc:492] T 82255aab37ba4f71b6245f39774e9cae P d020d94eb8224169bc14c10f564817f0: No bootstrap required, opened a new log
I20260810 03:53:16.758258 22367 tablet_bootstrap.cc:492] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281: No bootstrap required, opened a new log
I20260810 03:53:16.758283 22368 ts_tablet_manager.cc:1397] T 82255aab37ba4f71b6245f39774e9cae P d020d94eb8224169bc14c10f564817f0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:53:16.758455 22367 ts_tablet_manager.cc:1397] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 03:53:16.761557 22367 raft_consensus.cc:348] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } }
I20260810 03:53:16.761554 22368 raft_consensus.cc:348] T 82255aab37ba4f71b6245f39774e9cae P d020d94eb8224169bc14c10f564817f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } }
I20260810 03:53:16.761940 22367 raft_consensus.cc:374] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:53:16.761940 22368 raft_consensus.cc:374] T 82255aab37ba4f71b6245f39774e9cae P d020d94eb8224169bc14c10f564817f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:53:16.761992 22367 raft_consensus.cc:729] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8652e201d0ae4f138f7d7521871ac281, State: Initialized, Role: FOLLOWER
I20260810 03:53:16.761994 22368 raft_consensus.cc:729] T 82255aab37ba4f71b6245f39774e9cae P d020d94eb8224169bc14c10f564817f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d020d94eb8224169bc14c10f564817f0, State: Initialized, Role: FOLLOWER
I20260810 03:53:16.762437 22367 consensus_queue.cc:260] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } }
I20260810 03:53:16.762462 22368 consensus_queue.cc:260] T 82255aab37ba4f71b6245f39774e9cae P d020d94eb8224169bc14c10f564817f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } }
I20260810 03:53:16.763195 22108 heartbeater.cc:503] Master 127.27.153.126:45951 was elected leader, sending a full tablet report...
I20260810 03:53:16.763195 22344 heartbeater.cc:503] Master 127.27.153.126:45951 was elected leader, sending a full tablet report...
I20260810 03:53:16.763644 22367 ts_tablet_manager.cc:1428] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 03:53:16.763660 22368 ts_tablet_manager.cc:1428] T 82255aab37ba4f71b6245f39774e9cae P d020d94eb8224169bc14c10f564817f0: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260810 03:53:16.853138 22379 raft_consensus.cc:482] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:53:16.853375 22379 raft_consensus.cc:504] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } }
I20260810 03:53:16.854611 22379 leader_election.cc:288] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d020d94eb8224169bc14c10f564817f0 (127.27.153.66:39071), f28d5e87eeca486f9ab671c30f7e136f (127.27.153.65:38881)
I20260810 03:53:16.860025 22380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:16.855051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:53835 (local address 127.27.153.65:38881)
0810 03:53:16.855405 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:16.855409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:16.855423 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:53:16.855491 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:16.855493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:16.855551 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:16.855653 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:53:16.855928 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.856727 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.858538 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.859253 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.859371 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:16.859567 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:16.859792 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 03:53:16.859882 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":127,"threads_started":1}
I20260810 03:53:16.860024 22362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:16.855081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:53277 (local address 127.27.153.66:39071)
0810 03:53:16.855215 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:16.855219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:16.855552 (+ 333us) server_negotiation.cc:407] Connection header received
0810 03:53:16.855719 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:16.855722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:16.855787 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:16.855848 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:53:16.856092 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.856803 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.858538 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.859194 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.859355 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:16.859413 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:16.859800 (+ 387us) server_negotiation.cc:299] Negotiation successful
0810 03:53:16.859882 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 03:53:16.860028 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:16.854944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:38881 (local address 127.27.153.67:53835)
0810 03:53:16.855076 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:16.855103 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:16.855223 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:16.855678 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:16.855681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:16.855699 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:53:16.855805 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:16.855810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.856878 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.856883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:16.858354 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:16.858369 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.859380 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.859384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:16.859612 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:16.859665 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 03:53:16.859747 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 03:53:16.859867 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":170}
I20260810 03:53:16.860035 22381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:16.854949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:39071 (local address 127.27.153.67:53277)
0810 03:53:16.855425 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:16.855436 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:16.855581 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:16.855884 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:16.855887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:16.855904 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:53:16.855995 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:16.855999 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.856914 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.856917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:16.858354 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:16.858369 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.859380 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.859384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:16.859614 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:16.859665 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 03:53:16.859766 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 03:53:16.859853 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":251,"thread_start_us":135,"threads_started":1}
I20260810 03:53:16.860942 22027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82255aab37ba4f71b6245f39774e9cae" candidate_uuid: "8652e201d0ae4f138f7d7521871ac281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d020d94eb8224169bc14c10f564817f0" is_pre_election: true
I20260810 03:53:16.860898 21785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82255aab37ba4f71b6245f39774e9cae" candidate_uuid: "8652e201d0ae4f138f7d7521871ac281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f28d5e87eeca486f9ab671c30f7e136f" is_pre_election: true
I20260810 03:53:16.861274 21785 raft_consensus.cc:2454] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8652e201d0ae4f138f7d7521871ac281 in term 0.
I20260810 03:53:16.861274 22027 raft_consensus.cc:2454] T 82255aab37ba4f71b6245f39774e9cae P d020d94eb8224169bc14c10f564817f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8652e201d0ae4f138f7d7521871ac281 in term 0.
I20260810 03:53:16.861831 22150 leader_election.cc:302] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8652e201d0ae4f138f7d7521871ac281, d020d94eb8224169bc14c10f564817f0; no voters:
I20260810 03:53:16.862270 22379 raft_consensus.cc:2781] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:53:16.862368 22379 raft_consensus.cc:482] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:53:16.862424 22379 raft_consensus.cc:3037] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:53:16.863560 22379 raft_consensus.cc:504] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } }
I20260810 03:53:16.863929 22379 leader_election.cc:288] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281 [CANDIDATE]: Term 1 election: Requested vote from peers d020d94eb8224169bc14c10f564817f0 (127.27.153.66:39071), f28d5e87eeca486f9ab671c30f7e136f (127.27.153.65:38881)
I20260810 03:53:16.864349 21785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82255aab37ba4f71b6245f39774e9cae" candidate_uuid: "8652e201d0ae4f138f7d7521871ac281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f28d5e87eeca486f9ab671c30f7e136f"
I20260810 03:53:16.864351 22027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82255aab37ba4f71b6245f39774e9cae" candidate_uuid: "8652e201d0ae4f138f7d7521871ac281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d020d94eb8224169bc14c10f564817f0"
I20260810 03:53:16.864498 22027 raft_consensus.cc:3037] T 82255aab37ba4f71b6245f39774e9cae P d020d94eb8224169bc14c10f564817f0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:53:16.864497 21785 raft_consensus.cc:3037] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:53:16.865550 21785 raft_consensus.cc:2454] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8652e201d0ae4f138f7d7521871ac281 in term 1.
I20260810 03:53:16.865620 22027 raft_consensus.cc:2454] T 82255aab37ba4f71b6245f39774e9cae P d020d94eb8224169bc14c10f564817f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8652e201d0ae4f138f7d7521871ac281 in term 1.
I20260810 03:53:16.865912 22155 leader_election.cc:302] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8652e201d0ae4f138f7d7521871ac281, f28d5e87eeca486f9ab671c30f7e136f; no voters:
I20260810 03:53:16.866117 22379 raft_consensus.cc:2781] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:53:16.866349 22379 raft_consensus.cc:686] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281 [term 1 LEADER]: Becoming Leader. State: Replica: 8652e201d0ae4f138f7d7521871ac281, State: Running, Role: LEADER
I20260810 03:53:16.866739 22379 consensus_queue.cc:237] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } }
I20260810 03:53:16.870074 21519 catalog_manager.cc:5495] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8652e201d0ae4f138f7d7521871ac281 (127.27.153.67). New cstate: current_term: 1 leader_uuid: "8652e201d0ae4f138f7d7521871ac281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } health_report { overall_health: UNKNOWN } } }
I20260810 03:53:16.901736 22359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:16.898164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35176 (local address 127.27.153.67:37047)
0810 03:53:16.898293 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:16.898296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:16.898334 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:53:16.898414 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:16.898416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:16.898472 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:16.898552 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:53:16.898735 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.899248 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.900156 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:16.900390 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:16.900484 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:16.900850 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:53:16.901315 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:53:16.901422 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:16.901524 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 03:53:16.901589 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 03:53:16.908070 21785 raft_consensus.cc:1261] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Refusing update from remote peer 8652e201d0ae4f138f7d7521871ac281: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:53:16.908075 22027 raft_consensus.cc:1261] T 82255aab37ba4f71b6245f39774e9cae P d020d94eb8224169bc14c10f564817f0 [term 1 FOLLOWER]: Refusing update from remote peer 8652e201d0ae4f138f7d7521871ac281: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:53:16.908766 22382 consensus_queue.cc:1035] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:53:16.908955 22379 consensus_queue.cc:1035] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:53:16.916040 22387 mvcc.cc:205] Tried to move back new op lower bound from 7316824051327782912 to 7316824051169345536. Current Snapshot: MvccSnapshot[applied={T|T < 7316824051327782912}]
I20260810 03:53:16.917375 22388 mvcc.cc:205] Tried to move back new op lower bound from 7316824051327782912 to 7316824051169345536. Current Snapshot: MvccSnapshot[applied={T|T < 7316824051327782912}]
I20260810 03:53:16.925772 21517 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:55486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:53:16.925995 21517 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:55486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:53:16.928874 21517 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c: Sending DeleteTablet for 3 replicas of tablet 82255aab37ba4f71b6245f39774e9cae
I20260810 03:53:16.929734 21752 tablet_service.cc:1514] Processing DeleteTablet for tablet 82255aab37ba4f71b6245f39774e9cae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:53:16 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:34154
I20260810 03:53:16.929738 21998 tablet_service.cc:1514] Processing DeleteTablet for tablet 82255aab37ba4f71b6245f39774e9cae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:53:16 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:52746
I20260810 03:53:16.929806 22234 tablet_service.cc:1514] Processing DeleteTablet for tablet 82255aab37ba4f71b6245f39774e9cae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:53:16 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:35170
I20260810 03:53:16.930263 22400 tablet_replica.cc:330] T 82255aab37ba4f71b6245f39774e9cae P d020d94eb8224169bc14c10f564817f0: stopping tablet replica
I20260810 03:53:16.930312 22401 tablet_replica.cc:330] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281: stopping tablet replica
I20260810 03:53:16.930675 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 21596
I20260810 03:53:16.930683 22401 raft_consensus.cc:2229] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:53:16.930683 22400 raft_consensus.cc:2229] T 82255aab37ba4f71b6245f39774e9cae P d020d94eb8224169bc14c10f564817f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:53:16.933899 22400 raft_consensus.cc:2258] T 82255aab37ba4f71b6245f39774e9cae P d020d94eb8224169bc14c10f564817f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:53:16.936743 22401 raft_consensus.cc:2258] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:53:16.941807 22400 ts_tablet_manager.cc:1905] T 82255aab37ba4f71b6245f39774e9cae P d020d94eb8224169bc14c10f564817f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:53:16.943954 22401 ts_tablet_manager.cc:1905] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:53:16.945351 22400 ts_tablet_manager.cc:1918] T 82255aab37ba4f71b6245f39774e9cae P d020d94eb8224169bc14c10f564817f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:53:16.945606 22400 log.cc:1196] T 82255aab37ba4f71b6245f39774e9cae P d020d94eb8224169bc14c10f564817f0: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/82255aab37ba4f71b6245f39774e9cae
I20260810 03:53:16.946045 22400 ts_tablet_manager.cc:1939] T 82255aab37ba4f71b6245f39774e9cae P d020d94eb8224169bc14c10f564817f0: Deleting consensus metadata
I20260810 03:53:16.946743 22401 ts_tablet_manager.cc:1918] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:53:16.946885 22401 log.cc:1196] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/82255aab37ba4f71b6245f39774e9cae
I20260810 03:53:16.947232 22401 ts_tablet_manager.cc:1939] T 82255aab37ba4f71b6245f39774e9cae P 8652e201d0ae4f138f7d7521871ac281: Deleting consensus metadata
I20260810 03:53:16.947962 21481 catalog_manager.cc:4841] TS d020d94eb8224169bc14c10f564817f0 (127.27.153.66:39071): tablet 82255aab37ba4f71b6245f39774e9cae (table test-table [id=c218746065be46ffa37cc242f4e8a649]) successfully deleted
I20260810 03:53:16.958055 21483 catalog_manager.cc:4841] TS 8652e201d0ae4f138f7d7521871ac281 (127.27.153.67:37047): tablet 82255aab37ba4f71b6245f39774e9cae (table test-table [id=c218746065be46ffa37cc242f4e8a649]) successfully deleted
W20260810 03:53:16.963337 21487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.153.65:38881 (error 108)
I20260810 03:53:16.963728 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 21874
I20260810 03:53:16.967540 22357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:16.966992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:38881 (local address 127.0.0.1:34162)
0810 03:53:16.967253 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:16.967374 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:38881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260810 03:53:16.967808 21487 catalog_manager.cc:4584] TS f28d5e87eeca486f9ab671c30f7e136f (127.27.153.65:38881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82255aab37ba4f71b6245f39774e9cae: Network error: Client connection negotiation failed: client connection to 127.27.153.65:38881: connect: Connection refused (error 111)
I20260810 03:53:16.982784 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 22111
I20260810 03:53:16.997030 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 21452
I20260810 03:53:17.011296 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:45951
--webserver_interface=127.27.153.126
--webserver_port=34513
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:45951
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:53:17.129928 22404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:17.130362 22404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:17.130440 22404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:17.135752 22404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:53:17.135824 22404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:17.135843 22404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:17.135860 22404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:53:17.135874 22404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:53:17.141491 22404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:45951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:45951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=34513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:17.142673 22404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:17.143806 22404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:17.153059 22413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:17.153064 22415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:17.153065 22412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:17.153138 22404 server_base.cc:1034] running on GCE node
I20260810 03:53:17.153908 22404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:17.155141 22404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:17.156302 22404 hybrid_clock.cc:648] HybridClock initialized: now 1786333997156256 us; error 67 us; skew 500 ppm
Aug 10 03:53:17 dist-test-slave-pww0 krb5kdc[20456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333997, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:17.159286 22404 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:53:17.160321 22404 webserver.cc:460] Webserver started at http://127.27.153.126:34513/ using document root <none> and password file <none>
I20260810 03:53:17.160908 22404 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:17.160982 22404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:17.165102 22404 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260810 03:53:17.167428 22428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:17.168839 22404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 03:53:17.168974 22404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "6709cbea73ae4017848debfebb08f24c"
format_stamp: "Formatted at 2026-08-10 03:53:14 on dist-test-slave-pww0"
server_key: "9f89fede1da93d8154db3a2b3afd1c31"
server_key_iv: "0d1fabdae75bbd2557a31505da525274"
server_key_version: "encryptionkey@0"
I20260810 03:53:17.169589 22404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:17.182446 22404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:17.185678 22404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:17.185890 22404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:17.193197 22404 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:45951
I20260810 03:53:17.193475 22523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:45951 every 8 connection(s)
I20260810 03:53:17.194356 22404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:53:17.198871 22526 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:53:17.199050 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 22404
I20260810 03:53:17.199501 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:38881
--local_ip_for_outbound_sockets=127.27.153.65
--tserver_master_addrs=127.27.153.126:45951
--webserver_port=38537
--webserver_interface=127.27.153.65
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:53:17.200811 22526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c: Bootstrap starting.
I20260810 03:53:17.225605 22526 log.cc:824] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:17.240734 22526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:53:17.241220 22526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c: Bootstrap complete.
I20260810 03:53:17.246937 22526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709cbea73ae4017848debfebb08f24c" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45951 } }
I20260810 03:53:17.247546 22526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6709cbea73ae4017848debfebb08f24c, State: Initialized, Role: FOLLOWER
I20260810 03:53:17.248225 22526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709cbea73ae4017848debfebb08f24c" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45951 } }
I20260810 03:53:17.248391 22526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:53:17.248464 22526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:53:17.248566 22526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 2 FOLLOWER]: Advancing to term 3
I20260810 03:53:17.251361 22526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709cbea73ae4017848debfebb08f24c" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45951 } }
I20260810 03:53:17.251813 22526 leader_election.cc:302] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6709cbea73ae4017848debfebb08f24c; no voters:
I20260810 03:53:17.252192 22526 leader_election.cc:288] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 03:53:17.252388 22533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 3 FOLLOWER]: Leader election won for term 3
I20260810 03:53:17.252673 22533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [term 3 LEADER]: Becoming Leader. State: Replica: 6709cbea73ae4017848debfebb08f24c, State: Running, Role: LEADER
I20260810 03:53:17.253015 22533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709cbea73ae4017848debfebb08f24c" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45951 } }
I20260810 03:53:17.253753 22526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:53:17.254567 22534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6709cbea73ae4017848debfebb08f24c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709cbea73ae4017848debfebb08f24c" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45951 } } }
I20260810 03:53:17.254695 22534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [sys.catalog]: This master's current role is: LEADER
I20260810 03:53:17.254534 22536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6709cbea73ae4017848debfebb08f24c. Latest consensus state: current_term: 3 leader_uuid: "6709cbea73ae4017848debfebb08f24c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6709cbea73ae4017848debfebb08f24c" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 45951 } } }
I20260810 03:53:17.255179 22536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6709cbea73ae4017848debfebb08f24c [sys.catalog]: This master's current role is: LEADER
I20260810 03:53:17.255513 22546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:53:17.261057 22546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:53:17.262001 22546 catalog_manager.cc:1259] Loaded cluster ID: 20365940fafd4bd18913a6405bb087c3
I20260810 03:53:17.262081 22546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 03:53:17.263372 22546 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcd304bc9ad google::LogMessage::Fail() at ??:0
@ 0x7fcd304bf88a google::LogMessage::SendToLog() at ??:0
@ 0x7fcd304bc4de google::LogMessage::Flush() at ??:0
@ 0x7fcd304bda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcd3adfffdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fcd3ae00d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcd3adfd579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fcd3ae41640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fcd3270e3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcd3270ec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fcd327103f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fcd326ff512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcd33d436db start_thread at ??:0
@ 0x7fcd2efba61f clone at ??:0
W20260810 03:53:17.351092 22530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:17.351383 22530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:17.351449 22530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:17.355628 22530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:17.355695 22530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:17.355778 22530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:53:17.361379 22530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:38881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=38537
--tserver_master_addrs=127.27.153.126:45951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:17.362524 22530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:17.363621 22530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:17.373651 22604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:17.373834 22530 server_base.cc:1034] running on GCE node
W20260810 03:53:17.373661 22605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:17.373987 22609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:17.374490 22530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:17.375195 22530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:17.376362 22530 hybrid_clock.cc:648] HybridClock initialized: now 1786333997376322 us; error 82 us; skew 500 ppm
Aug 10 03:53:17 dist-test-slave-pww0 krb5kdc[20456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333997, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:17.379276 22530 init.cc:377] Logged in from keytab as kudu/127.27.153.65@KRBTEST.COM (short username kudu)
I20260810 03:53:17.380295 22530 webserver.cc:460] Webserver started at http://127.27.153.65:38537/ using document root <none> and password file <none>
I20260810 03:53:17.380857 22530 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:17.380923 22530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:17.385098 22530 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:53:17.387775 22618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:17.389588 22530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:53:17.389734 22530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "f28d5e87eeca486f9ab671c30f7e136f"
format_stamp: "Formatted at 2026-08-10 03:53:14 on dist-test-slave-pww0"
server_key: "6c9c1f5ba883dad47917d6f5eaddc93d"
server_key_iv: "9c78c69df8871b6c9c0ea1bfc112b776"
server_key_version: "encryptionkey@0"
I20260810 03:53:17.390528 22530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:17.405681 22530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:17.409715 22530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:17.409965 22530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:17.410818 22530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:53:17.412396 22626 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 03:53:17.416791 22530 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 03:53:17.416896 22530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260810 03:53:17.416993 22530 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 03:53:17.419523 22530 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 03:53:17.419612 22530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.003s
I20260810 03:53:17.419878 22626 tablet_bootstrap.cc:492] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f: Bootstrap starting.
I20260810 03:53:17.432288 22530 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:38881
I20260810 03:53:17.432327 22820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:38881 every 8 connection(s)
I20260810 03:53:17.433681 22530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:53:17.438823 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 22530
I20260810 03:53:17.439283 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:39071
--local_ip_for_outbound_sockets=127.27.153.66
--tserver_master_addrs=127.27.153.126:45951
--webserver_port=38735
--webserver_interface=127.27.153.66
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:53:17.440625 22626 log.cc:824] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:17.450366 22626 tablet_bootstrap.cc:492] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:53:17.450846 22626 tablet_bootstrap.cc:492] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f: Bootstrap complete.
I20260810 03:53:17.452343 22626 ts_tablet_manager.cc:1397] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f: Time spent bootstrapping tablet: real 0.033s user 0.015s sys 0.016s
I20260810 03:53:17.455755 22626 raft_consensus.cc:348] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } }
I20260810 03:53:17.456363 22626 raft_consensus.cc:729] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f28d5e87eeca486f9ab671c30f7e136f, State: Initialized, Role: FOLLOWER
I20260810 03:53:17.456941 22626 consensus_queue.cc:260] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } }
I20260810 03:53:17.458071 22626 ts_tablet_manager.cc:1428] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260810 03:53:17.495819 22827 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:53:17.436247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.65:57759)
0810 03:53:17.436942 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:17.437007 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:17.438149 (+ 1142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:17.494711 (+ 56562us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":188,"threads_started":1}
W20260810 03:53:17.496444 22821 heartbeater.cc:644] Failed to heartbeat to 127.27.153.126:45951 (0 consecutive failures): Network error: Failed to ping master at 127.27.153.126:45951: Client connection negotiation failed: client connection to 127.27.153.126:45951: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 03:53:17.497332 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:17.497038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.65:42541)
0810 03:53:17.497175 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:17.497254 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 03:53:17.497975 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:17.497757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.65:55301)
0810 03:53:17.497866 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:17.497917 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 03:53:17.498163 22821 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:53:17.564431 22831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:17.564738 22831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:17.564783 22831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:17.569067 22831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:17.569149 22831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:17.569212 22831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:53:17.575065 22831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:39071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=38735
--tserver_master_addrs=127.27.153.126:45951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:17.576260 22831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:17.577301 22831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:17.587419 22848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:17.587436 22847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:17.587436 22850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:17.587777 22831 server_base.cc:1034] running on GCE node
I20260810 03:53:17.588270 22831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:17.588989 22831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:17.590159 22831 hybrid_clock.cc:648] HybridClock initialized: now 1786333997590137 us; error 39 us; skew 500 ppm
Aug 10 03:53:17 dist-test-slave-pww0 krb5kdc[20456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333997, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:17.593199 22831 init.cc:377] Logged in from keytab as kudu/127.27.153.66@KRBTEST.COM (short username kudu)
I20260810 03:53:17.594215 22831 webserver.cc:460] Webserver started at http://127.27.153.66:38735/ using document root <none> and password file <none>
I20260810 03:53:17.594838 22831 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:17.594919 22831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:17.599313 22831 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:53:17.601612 22861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:17.603009 22831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:53:17.603153 22831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "d020d94eb8224169bc14c10f564817f0"
format_stamp: "Formatted at 2026-08-10 03:53:15 on dist-test-slave-pww0"
server_key: "c720c48cd55f9e3224da960677eb8b1f"
server_key_iv: "c8853d6c6ae3ab4d2c44050b8cddf8a5"
server_key_version: "encryptionkey@0"
I20260810 03:53:17.603834 22831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:17.616765 22831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:17.620287 22831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:17.620491 22831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:17.621125 22831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:53:17.622020 22831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:53:17.622069 22831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:17.622128 22831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:53:17.622154 22831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:17.631539 22831 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:39071
I20260810 03:53:17.631596 23063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:39071 every 8 connection(s)
I20260810 03:53:17.632655 22831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:53:17.636786 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:17.634976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.66:57969)
0810 03:53:17.635541 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:17.636014 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":121,"threads_started":1}
W20260810 03:53:17.637264 23064 heartbeater.cc:644] Failed to heartbeat to 127.27.153.126:45951 (0 consecutive failures): Network error: Failed to ping master at 127.27.153.126:45951: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
I20260810 03:53:17.637902 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:17.637665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.66:34593)
0810 03:53:17.637790 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:17.637842 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 03:53:17.638489 23066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:17.638292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.66:50847)
0810 03:53:17.638389 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:17.638436 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 03:53:17.638636 23064 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 03:53:17.641944 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 22831
I20260810 03:53:17.642365 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:37047
--local_ip_for_outbound_sockets=127.27.153.67
--tserver_master_addrs=127.27.153.126:45951
--webserver_port=41215
--webserver_interface=127.27.153.67
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:53:17.758966 23071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:17.759328 23071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:17.759382 23071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:17.763599 23071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:17.763679 23071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:17.763741 23071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:53:17.769176 23071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:37047
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=41215
--tserver_master_addrs=127.27.153.126:45951
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:17.770318 23071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:17.771377 23071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:17.781188 23080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:17.781200 23085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:17.781216 23081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:17.781558 23071 server_base.cc:1034] running on GCE node
I20260810 03:53:17.781988 23071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:17.782651 23071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:17.783804 23071 hybrid_clock.cc:648] HybridClock initialized: now 1786333997783784 us; error 33 us; skew 500 ppm
Aug 10 03:53:17 dist-test-slave-pww0 krb5kdc[20456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333997, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:17.786573 23071 init.cc:377] Logged in from keytab as kudu/127.27.153.67@KRBTEST.COM (short username kudu)
I20260810 03:53:17.787582 23071 webserver.cc:460] Webserver started at http://127.27.153.67:41215/ using document root <none> and password file <none>
I20260810 03:53:17.788141 23071 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:17.788206 23071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:17.792285 23071 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260810 03:53:17.794525 23097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:17.796000 23071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:53:17.796139 23071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "8652e201d0ae4f138f7d7521871ac281"
format_stamp: "Formatted at 2026-08-10 03:53:15 on dist-test-slave-pww0"
server_key: "5c1124ab0fa24dc45b41c83bdac19eb9"
server_key_iv: "b3a6cb03c22adb7684d5a6cf12761caa"
server_key_version: "encryptionkey@0"
I20260810 03:53:17.796721 23071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:17.825352 23071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:17.828691 23071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:17.828900 23071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:17.829577 23071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:53:17.830544 23071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:53:17.830597 23071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:17.830657 23071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:53:17.830687 23071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:17.840322 23071 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:37047
I20260810 03:53:17.840413 23302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:37047 every 8 connection(s)
I20260810 03:53:17.841473 23071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:53:17.845518 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:17.843768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.67:38673)
0810 03:53:17.844304 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:17.844759 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":126,"threads_started":1}
W20260810 03:53:17.846043 23306 heartbeater.cc:644] Failed to heartbeat to 127.27.153.126:45951 (0 consecutive failures): Network error: Failed to ping master at 127.27.153.126:45951: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
I20260810 03:53:17.846670 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:17.846447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.67:45445)
0810 03:53:17.846558 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:17.846611 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 03:53:17.847298 23309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:17.847107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.67:49185)
0810 03:53:17.847193 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:17.847235 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 03:53:17.847450 23306 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 03:53:17.850915 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 23071
I20260810 03:53:18.499226 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:18.498743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.65:50459)
0810 03:53:18.499030 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:18.499096 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":82,"threads_started":1}
I20260810 03:53:18.639614 23313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:18.639177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.66:60961)
0810 03:53:18.639418 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:18.639506 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":72,"threads_started":1}
I20260810 03:53:18.848594 23315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:18.848094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.67:34065)
0810 03:53:18.848400 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:18.848465 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":105,"threads_started":1}
I20260810 03:53:18.969491 23316 raft_consensus.cc:482] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:53:18.969739 23316 raft_consensus.cc:504] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } }
I20260810 03:53:18.970961 23316 leader_election.cc:288] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8652e201d0ae4f138f7d7521871ac281 (127.27.153.67:37047), d020d94eb8224169bc14c10f564817f0 (127.27.153.66:39071)
Aug 10 03:53:18 dist-test-slave-pww0 krb5kdc[20456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333997, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.67@KRBTEST.COM
Aug 10 03:53:18 dist-test-slave-pww0 krb5kdc[20456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786333997, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.66@KRBTEST.COM
I20260810 03:53:18.996088 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:18.971198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:37047 (local address 127.27.153.65:56735)
0810 03:53:18.971591 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:18.971604 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:18.971719 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:18.973390 (+ 1671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:18.973393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:18.973778 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:18.974237 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:18.974253 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:18.975562 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:18.975565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:18.976574 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:18.976585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:18.977326 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:18.977330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:18.977575 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:18.986404 (+ 8829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:18.986437 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:18.989796 (+ 3359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:18.993153 (+ 3357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:18.993161 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:18.993179 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:18.993587 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:18.994036 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:18.994040 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:18.994042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:18.994227 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:18.994988 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:18.994996 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:18.995276 (+ 280us) client_negotiation.cc:769] Sending connection context
0810 03:53:18.995525 (+ 249us) client_negotiation.cc:240] Negotiation successful
0810 03:53:18.995879 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":142,"spinlock_wait_cycles":10496,"thread_start_us":91,"threads_started":1}
I20260810 03:53:18.996114 23329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:18.971759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:56735 (local address 127.27.153.67:37047)
0810 03:53:18.972075 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:18.972095 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:18.972128 (+ 33us) server_negotiation.cc:407] Connection header received
0810 03:53:18.972843 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:18.972849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:18.973156 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:18.973422 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:18.974412 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:18.975392 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:18.976704 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:18.977201 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:18.977356 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:18.990161 (+ 12805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:18.990169 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:18.990190 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:18.990232 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:18.992967 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:18.993738 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:18.993746 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:18.993752 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:18.993927 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:18.994329 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:18.994333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:18.994335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:18.994863 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:18.995002 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:18.995671 (+ 669us) server_negotiation.cc:299] Negotiation successful
0810 03:53:18.995954 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":60,"threads_started":1}
I20260810 03:53:18.999073 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:18.971202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:39071 (local address 127.27.153.65:37891)
0810 03:53:18.971392 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:18.971407 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:18.971553 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:18.973243 (+ 1690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:18.973249 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:18.973766 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:18.974237 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:18.974253 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:18.975565 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:18.975569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:18.976573 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:18.976584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:18.977326 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:18.977330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:18.977553 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:18.986739 (+ 9186us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:18.986756 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:18.989796 (+ 3040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:18.996887 (+ 7091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:18.996891 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:18.996893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:18.997131 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:18.997682 (+ 551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:18.997686 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:18.997688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:18.997780 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:18.998543 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:18.998546 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:18.998655 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 03:53:18.998744 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 03:53:18.998896 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":135}
W20260810 03:53:18.999950 22620 leader_election.cc:341] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8652e201d0ae4f138f7d7521871ac281 (127.27.153.67:37047): Not found: Tablet not found: 82255aab37ba4f71b6245f39774e9cae
I20260810 03:53:19.000399 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:18.971412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:37891 (local address 127.27.153.66:39071)
0810 03:53:18.971714 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:18.971740 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:18.971777 (+ 37us) server_negotiation.cc:407] Connection header received
0810 03:53:18.972467 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:18.972473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:18.972842 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:18.973155 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:18.974413 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:18.975393 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:18.976728 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:18.977199 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:18.977356 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:18.990161 (+ 12805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:18.990169 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:18.990190 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:18.990232 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:18.996731 (+ 6499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:18.997288 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:18.997295 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:18.997301 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:18.997553 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:18.997898 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:18.997901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:18.997903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:18.998418 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:18.999739 (+ 1321us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:18.999899 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 03:53:19.000225 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":99,"threads_started":1}
W20260810 03:53:19.002857 22619 leader_election.cc:341] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d020d94eb8224169bc14c10f564817f0 (127.27.153.66:39071): Not found: Tablet not found: 82255aab37ba4f71b6245f39774e9cae
I20260810 03:53:19.002964 22619 leader_election.cc:302] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f28d5e87eeca486f9ab671c30f7e136f; no voters: 8652e201d0ae4f138f7d7521871ac281, d020d94eb8224169bc14c10f564817f0
I20260810 03:53:19.003314 23316 raft_consensus.cc:2726] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:53:19.500337 23323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:19.500030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.65:44315)
0810 03:53:19.500171 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:19.500241 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 03:53:19.641010 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:19.640443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.66:53415)
0810 03:53:19.640799 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:19.640887 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":85,"threads_started":1}
I20260810 03:53:19.849915 23331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:19.849440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.67:58249)
0810 03:53:19.849751 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:19.849819 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":83,"threads_started":1}
I20260810 03:53:20.501638 23332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:20.501101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.65:47187)
0810 03:53:20.501439 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:20.501519 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":100,"threads_started":1}
I20260810 03:53:20.507853 23335 raft_consensus.cc:482] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:53:20.507987 23335 raft_consensus.cc:504] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } }
I20260810 03:53:20.508478 23335 leader_election.cc:288] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8652e201d0ae4f138f7d7521871ac281 (127.27.153.67:37047), d020d94eb8224169bc14c10f564817f0 (127.27.153.66:39071)
W20260810 03:53:20.509301 22619 leader_election.cc:341] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d020d94eb8224169bc14c10f564817f0 (127.27.153.66:39071): Not found: Tablet not found: 82255aab37ba4f71b6245f39774e9cae
W20260810 03:53:20.509418 22620 leader_election.cc:341] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8652e201d0ae4f138f7d7521871ac281 (127.27.153.67:37047): Not found: Tablet not found: 82255aab37ba4f71b6245f39774e9cae
I20260810 03:53:20.509479 22620 leader_election.cc:302] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f28d5e87eeca486f9ab671c30f7e136f; no voters: 8652e201d0ae4f138f7d7521871ac281, d020d94eb8224169bc14c10f564817f0
I20260810 03:53:20.509630 23335 raft_consensus.cc:2726] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:53:20.642629 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:20.641988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.66:37887)
0810 03:53:20.642350 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:20.642424 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":123,"threads_started":1}
I20260810 03:53:20.851302 23340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:20.850834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.67:52223)
0810 03:53:20.851131 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:20.851201 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":91,"threads_started":1}
I20260810 03:53:21.502951 23342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:21.502459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.65:48199)
0810 03:53:21.502781 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:21.502851 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":86,"threads_started":1}
I20260810 03:53:21.643918 23346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:21.643484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.66:33827)
0810 03:53:21.643743 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:21.643813 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":74,"threads_started":1}
I20260810 03:53:21.852730 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:21.852272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.67:49231)
0810 03:53:21.852562 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:21.852627 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":84,"threads_started":1}
I20260810 03:53:22.504523 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:22.503977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.65:57093)
0810 03:53:22.504349 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:22.504423 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":82,"threads_started":1}
I20260810 03:53:22.645236 23355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:22.644771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.66:53365)
0810 03:53:22.645065 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:22.645130 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":81,"threads_started":1}
I20260810 03:53:22.854189 23356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:22.853738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.67:32885)
0810 03:53:22.854022 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:22.854088 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":82,"threads_started":1}
I20260810 03:53:23.505790 23357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:23.505324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.65:47863)
0810 03:53:23.505620 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:23.505682 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":80,"threads_started":1}
I20260810 03:53:23.646529 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:23.646083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.66:51147)
0810 03:53:23.646359 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:23.646427 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
I20260810 03:53:23.850126 23359 raft_consensus.cc:482] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:53:23.850314 23359 raft_consensus.cc:504] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } }
I20260810 03:53:23.850778 23359 leader_election.cc:288] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8652e201d0ae4f138f7d7521871ac281 (127.27.153.67:37047), d020d94eb8224169bc14c10f564817f0 (127.27.153.66:39071)
W20260810 03:53:23.851701 22620 leader_election.cc:341] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8652e201d0ae4f138f7d7521871ac281 (127.27.153.67:37047): Not found: Tablet not found: 82255aab37ba4f71b6245f39774e9cae
W20260810 03:53:23.851842 22619 leader_election.cc:341] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d020d94eb8224169bc14c10f564817f0 (127.27.153.66:39071): Not found: Tablet not found: 82255aab37ba4f71b6245f39774e9cae
I20260810 03:53:23.851940 22619 leader_election.cc:302] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f28d5e87eeca486f9ab671c30f7e136f; no voters: 8652e201d0ae4f138f7d7521871ac281, d020d94eb8224169bc14c10f564817f0
I20260810 03:53:23.852128 23359 raft_consensus.cc:2726] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:53:23.855363 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:23.854944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.67:47245)
0810 03:53:23.855211 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:23.855272 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":95,"threads_started":1}
I20260810 03:53:24.507073 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:24.506614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.65:43623)
0810 03:53:24.506902 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:24.506971 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":81,"threads_started":1}
I20260810 03:53:24.647910 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:24.647356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.66:45643)
0810 03:53:24.647727 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:24.647803 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":98,"threads_started":1}
I20260810 03:53:24.856691 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:24.856141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.67:41325)
0810 03:53:24.856518 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:24.856587 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":83,"threads_started":1}
I20260810 03:53:25.508387 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:25.507971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.65:51451)
0810 03:53:25.508226 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:25.508284 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":78,"threads_started":1}
I20260810 03:53:25.649327 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:25.648857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.66:39383)
0810 03:53:25.649146 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:25.649212 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":86,"threads_started":1}
I20260810 03:53:25.858000 23371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:25.857519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.67:46373)
0810 03:53:25.857830 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:25.857901 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":79,"threads_started":1}
I20260810 03:53:26.509630 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:26.509148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.65:40023)
0810 03:53:26.509460 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:26.509527 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":96,"threads_started":1}
I20260810 03:53:26.650630 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:26.650185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.66:42229)
0810 03:53:26.650463 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:26.650529 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":92,"threads_started":1}
I20260810 03:53:26.859383 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:26.858862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.67:48037)
0810 03:53:26.859198 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:26.859268 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":103,"threads_started":1}
I20260810 03:53:27.017498 23385 raft_consensus.cc:482] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:53:27.017652 23385 raft_consensus.cc:504] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } }
I20260810 03:53:27.018113 23385 leader_election.cc:288] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8652e201d0ae4f138f7d7521871ac281 (127.27.153.67:37047), d020d94eb8224169bc14c10f564817f0 (127.27.153.66:39071)
W20260810 03:53:27.018885 22619 leader_election.cc:341] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d020d94eb8224169bc14c10f564817f0 (127.27.153.66:39071): Not found: Tablet not found: 82255aab37ba4f71b6245f39774e9cae
W20260810 03:53:27.018986 22620 leader_election.cc:341] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8652e201d0ae4f138f7d7521871ac281 (127.27.153.67:37047): Not found: Tablet not found: 82255aab37ba4f71b6245f39774e9cae
I20260810 03:53:27.019075 22620 leader_election.cc:302] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f28d5e87eeca486f9ab671c30f7e136f; no voters: 8652e201d0ae4f138f7d7521871ac281, d020d94eb8224169bc14c10f564817f0
I20260810 03:53:27.019208 23385 raft_consensus.cc:2726] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:53:27.510972 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:27.510519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.65:39909)
0810 03:53:27.510805 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:27.510868 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
I20260810 03:53:27.652179 23387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:27.651688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.66:53775)
0810 03:53:27.652003 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:27.652074 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":86,"threads_started":1}
I20260810 03:53:27.860738 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:27.860272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.67:59309)
0810 03:53:27.860570 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:27.860637 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":102,"threads_started":1}
I20260810 03:53:28.512367 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:28.511892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.65:33291)
0810 03:53:28.512179 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:28.512246 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":84,"threads_started":1}
I20260810 03:53:28.653622 23392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:28.653078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.66:37019)
0810 03:53:28.653418 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:28.653508 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":84,"threads_started":1}
I20260810 03:53:28.862126 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:28.861615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.67:58777)
0810 03:53:28.861911 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:28.861988 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":79,"threads_started":1}
I20260810 03:53:29.513671 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:29.513226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.65:40139)
0810 03:53:29.513501 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:29.513570 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":83,"threads_started":1}
I20260810 03:53:29.655078 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:29.654607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.66:36385)
0810 03:53:29.654905 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:29.654977 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":93,"threads_started":1}
I20260810 03:53:29.863488 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:29.863024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.67:54279)
0810 03:53:29.863302 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:29.863367 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":89,"threads_started":1}
I20260810 03:53:30.514973 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:30.514470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.65:43581)
0810 03:53:30.514806 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:30.514872 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":99,"threads_started":1}
I20260810 03:53:30.656394 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:30.655937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.66:60835)
0810 03:53:30.656227 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:30.656294 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":91,"threads_started":1}
I20260810 03:53:30.864823 23405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:30.864348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.67:44605)
0810 03:53:30.864653 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:30.864722 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":86,"threads_started":1}
I20260810 03:53:31.516343 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:31.515885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.65:49873)
0810 03:53:31.516172 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:31.516241 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20260810 03:53:31.657887 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:31.657348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.66:39143)
0810 03:53:31.657689 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:31.657782 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":134,"threads_started":1}
I20260810 03:53:31.795747 23409 raft_consensus.cc:482] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:53:31.795909 23409 raft_consensus.cc:504] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8652e201d0ae4f138f7d7521871ac281" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 37047 } } peers { permanent_uuid: "d020d94eb8224169bc14c10f564817f0" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 39071 } } peers { permanent_uuid: "f28d5e87eeca486f9ab671c30f7e136f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38881 } }
I20260810 03:53:31.796422 23409 leader_election.cc:288] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8652e201d0ae4f138f7d7521871ac281 (127.27.153.67:37047), d020d94eb8224169bc14c10f564817f0 (127.27.153.66:39071)
W20260810 03:53:31.797194 22619 leader_election.cc:341] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d020d94eb8224169bc14c10f564817f0 (127.27.153.66:39071): Not found: Tablet not found: 82255aab37ba4f71b6245f39774e9cae
W20260810 03:53:31.797338 22620 leader_election.cc:341] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8652e201d0ae4f138f7d7521871ac281 (127.27.153.67:37047): Not found: Tablet not found: 82255aab37ba4f71b6245f39774e9cae
I20260810 03:53:31.797400 22620 leader_election.cc:302] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f28d5e87eeca486f9ab671c30f7e136f; no voters: 8652e201d0ae4f138f7d7521871ac281, d020d94eb8224169bc14c10f564817f0
I20260810 03:53:31.797539 23409 raft_consensus.cc:2726] T 82255aab37ba4f71b6245f39774e9cae P f28d5e87eeca486f9ab671c30f7e136f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:53:31.866158 23413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:31.865659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.67:34925)
0810 03:53:31.865985 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:31.866054 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":88,"threads_started":1}
I20260810 03:53:32.517803 23417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:32.517357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.65:56817)
0810 03:53:32.517618 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:32.517680 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":80,"threads_started":1}
I20260810 03:53:32.659547 23418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:32.659009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.66:45093)
0810 03:53:32.659340 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:32.659408 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":111,"threads_started":1}
I20260810 03:53:32.851994 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 22530
I20260810 03:53:32.871080 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:32.868936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:45951 (local address 127.27.153.67:53219)
0810 03:53:32.870852 (+ 1916us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:32.870954 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:45951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1795,"thread_start_us":1484,"threads_started":1}
I20260810 03:53:32.876144 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 22831
I20260810 03:53:32.895668 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 23071
2026-08-10T03:53:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19772 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:53:32 dist-test-slave-pww0 krb5kdc[23426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:53:32 dist-test-slave-pww0 krb5kdc[23426](info): set up 2 sockets
Aug 10 03:53:32 dist-test-slave-pww0 krb5kdc[23426](info): commencing operation
krb5kdc: starting...
W20260810 03:53:34.107242 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.008s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:53:34 dist-test-slave-pww0 krb5kdc[23426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:53:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:53:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:53:34.238445 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:42371
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:43087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:42371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:53:34.358794 23457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:34.359128 23457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:34.359192 23457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:34.363552 23457 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 03:53:34.363622 23457 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 03:53:34.363644 23457 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 03:53:34.363662 23457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:53:34.363687 23457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:34.363723 23457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:34.363754 23457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:53:34.363775 23457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:53:34.369448 23457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:43087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:42371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:42371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:34.370644 23457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:34.371757 23457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:34.380381 23463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:34.380388 23467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:34.380445 23457 server_base.cc:1034] running on GCE node
W20260810 03:53:34.380388 23464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:34.381289 23457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:34.382457 23457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:34.383641 23457 hybrid_clock.cc:648] HybridClock initialized: now 1786334014383619 us; error 34 us; skew 500 ppm
Aug 10 03:53:34 dist-test-slave-pww0 krb5kdc[23426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334014, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:34.386494 23457 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:53:34.387419 23457 webserver.cc:460] Webserver started at http://127.27.153.126:41403/ using document root <none> and password file <none>
I20260810 03:53:34.388002 23457 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:34.388064 23457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:34.388247 23457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:53:34.390309 23457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "356ae6557eff400d8b1de99f512af1be"
format_stamp: "Formatted at 2026-08-10 03:53:34 on dist-test-slave-pww0"
server_key: "ff91e49bdb94950e2014f72683062e60"
server_key_iv: "cc8f5d442a8dbc83e604b1f755b7b470"
server_key_version: "encryptionkey@0"
I20260810 03:53:34.390848 23457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "356ae6557eff400d8b1de99f512af1be"
format_stamp: "Formatted at 2026-08-10 03:53:34 on dist-test-slave-pww0"
server_key: "ff91e49bdb94950e2014f72683062e60"
server_key_iv: "cc8f5d442a8dbc83e604b1f755b7b470"
server_key_version: "encryptionkey@0"
I20260810 03:53:34.394567 23457 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 03:53:34.397190 23482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:34.398448 23457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 03:53:34.398562 23457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "356ae6557eff400d8b1de99f512af1be"
format_stamp: "Formatted at 2026-08-10 03:53:34 on dist-test-slave-pww0"
server_key: "ff91e49bdb94950e2014f72683062e60"
server_key_iv: "cc8f5d442a8dbc83e604b1f755b7b470"
server_key_version: "encryptionkey@0"
I20260810 03:53:34.398681 23457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:34.414831 23457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:34.418148 23457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:34.418339 23457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:34.425752 23457 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:42371
I20260810 03:53:34.426045 23575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:42371 every 8 connection(s)
I20260810 03:53:34.426785 23457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:53:34.427871 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 23457
I20260810 03:53:34.428004 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:53:34.428314 28261 external_mini_cluster.cc:1428] Setting key d5bbceb1f1bebf240a3edd0ca92c044a
I20260810 03:53:34.430227 23576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:53:34.436971 23576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be: Bootstrap starting.
Aug 10 03:53:34 dist-test-slave-pww0 krb5kdc[23426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:53:34.439661 23576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be: Neither blocks nor log segments found. Creating new log.
I20260810 03:53:34.440443 23576 log.cc:824] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:34.442744 23576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be: No bootstrap required, opened a new log
I20260810 03:53:34.443346 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:34.429824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55578 (local address 127.27.153.126:42371)
0810 03:53:34.430277 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:34.430313 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:34.430363 (+ 50us) server_negotiation.cc:407] Connection header received
0810 03:53:34.431078 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:34.431096 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:34.431438 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:34.431802 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:34.432698 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:34.433760 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:34.434931 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:34.435415 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:34.435614 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:34.438520 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:34.438525 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:34.438539 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:34.438571 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:34.440282 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:34.440784 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:34.440789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:34.440793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:34.440931 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:34.441210 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:34.441213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:34.441215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:34.441637 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:34.441784 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:34.442125 (+ 341us) server_negotiation.cc:299] Negotiation successful
0810 03:53:34.442410 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":144,"threads_started":1}
I20260810 03:53:34.447084 23576 raft_consensus.cc:348] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356ae6557eff400d8b1de99f512af1be" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42371 } }
I20260810 03:53:34.447327 23576 raft_consensus.cc:374] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:53:34.447378 23576 raft_consensus.cc:729] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 356ae6557eff400d8b1de99f512af1be, State: Initialized, Role: FOLLOWER
I20260810 03:53:34.447839 23576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356ae6557eff400d8b1de99f512af1be" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42371 } }
I20260810 03:53:34.447968 23576 raft_consensus.cc:388] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:53:34.448030 23576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:53:34.448124 23576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:53:34.449074 23576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356ae6557eff400d8b1de99f512af1be" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42371 } }
I20260810 03:53:34.449411 23576 leader_election.cc:302] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 356ae6557eff400d8b1de99f512af1be; no voters:
I20260810 03:53:34.449710 23576 leader_election.cc:288] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:53:34.449824 23584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:53:34.450083 23584 raft_consensus.cc:686] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 1 LEADER]: Becoming Leader. State: Replica: 356ae6557eff400d8b1de99f512af1be, State: Running, Role: LEADER
I20260810 03:53:34.450469 23584 consensus_queue.cc:237] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356ae6557eff400d8b1de99f512af1be" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42371 } }
I20260810 03:53:34.450969 23576 sys_catalog.cc:564] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:53:34.452128 23588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 356ae6557eff400d8b1de99f512af1be. Latest consensus state: current_term: 1 leader_uuid: "356ae6557eff400d8b1de99f512af1be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356ae6557eff400d8b1de99f512af1be" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42371 } } }
I20260810 03:53:34.452265 23588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [sys.catalog]: This master's current role is: LEADER
I20260810 03:53:34.452653 23599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:53:34.452653 23587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "356ae6557eff400d8b1de99f512af1be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356ae6557eff400d8b1de99f512af1be" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42371 } } }
I20260810 03:53:34.452795 23587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [sys.catalog]: This master's current role is: LEADER
I20260810 03:53:34.456125 23599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:53:34.461896 23599 catalog_manager.cc:1347] Generated new cluster ID: b77c67349c6342e4932dc89ba6409c7d
I20260810 03:53:34.461992 23599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:53:34.479998 23599 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:53:34.481077 23599 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:53:34.489840 23599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be: Generated new TSK 0
I20260810 03:53:34.490731 23599 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.153.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
I20260810 03:53:34.532641 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:42371
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:43087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:53:34.648502 23618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:34.648872 23618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:34.648921 23618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:34.653292 23618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:34.653385 23618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:34.653455 23618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:53:34.659004 23618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:43087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:42371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:34.660223 23618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:34.661363 23618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:34.671908 23625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:34.671952 23629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:34.671945 23627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:34.672259 23618 server_base.cc:1034] running on GCE node
I20260810 03:53:34.672725 23618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:34.673403 23618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:34.674572 23618 hybrid_clock.cc:648] HybridClock initialized: now 1786334014674532 us; error 54 us; skew 500 ppm
Aug 10 03:53:34 dist-test-slave-pww0 krb5kdc[23426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334014, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:34.677515 23618 init.cc:377] Logged in from keytab as kudu/127.27.153.65@KRBTEST.COM (short username kudu)
I20260810 03:53:34.678488 23618 webserver.cc:460] Webserver started at http://127.27.153.65:38379/ using document root <none> and password file <none>
I20260810 03:53:34.679041 23618 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:34.679104 23618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:34.679333 23618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:53:34.681511 23618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547"
format_stamp: "Formatted at 2026-08-10 03:53:34 on dist-test-slave-pww0"
server_key: "ffc2269d8c50241c17c7978e2a131963"
server_key_iv: "91b3579162793ec72b95edd75cadfc68"
server_key_version: "encryptionkey@0"
I20260810 03:53:34.682066 23618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547"
format_stamp: "Formatted at 2026-08-10 03:53:34 on dist-test-slave-pww0"
server_key: "ffc2269d8c50241c17c7978e2a131963"
server_key_iv: "91b3579162793ec72b95edd75cadfc68"
server_key_version: "encryptionkey@0"
I20260810 03:53:34.685933 23618 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:53:34.688855 23648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:34.690307 23618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:53:34.690460 23618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547"
format_stamp: "Formatted at 2026-08-10 03:53:34 on dist-test-slave-pww0"
server_key: "ffc2269d8c50241c17c7978e2a131963"
server_key_iv: "91b3579162793ec72b95edd75cadfc68"
server_key_version: "encryptionkey@0"
I20260810 03:53:34.690588 23618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:34.707433 23618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:34.710520 23618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:34.710721 23618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:34.711361 23618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:53:34.712260 23618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:53:34.712312 23618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:34.712369 23618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:53:34.712394 23618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:34.721076 23618 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:38975
I20260810 03:53:34.721098 23843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:38975 every 8 connection(s)
I20260810 03:53:34.722043 23618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:53:34.730774 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 23618
I20260810 03:53:34.730923 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:53:34.731220 28261 external_mini_cluster.cc:1428] Setting key d5e80cb7a67a0e363dedbda400393349
Aug 10 03:53:34 dist-test-slave-pww0 krb5kdc[23426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334014, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:53:34.735942 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:34.723881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:40769 (local address 127.27.153.126:42371)
0810 03:53:34.724072 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:34.724076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:34.725052 (+ 976us) server_negotiation.cc:407] Connection header received
0810 03:53:34.725918 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:34.725921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:34.725968 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:34.726031 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:34.727234 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:34.727919 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:34.729090 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:34.729317 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:34.729391 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:34.732169 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:34.732172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:34.732175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:34.732201 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:34.733677 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:34.734311 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:34.734314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:34.734315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:34.734398 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:34.734748 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:34.734750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:34.734751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:34.734967 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:34.735070 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:34.735628 (+ 558us) server_negotiation.cc:299] Negotiation successful
0810 03:53:34.735774 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 03:53:34.736765 23846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:34.724257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.65:40769)
0810 03:53:34.724907 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:34.724948 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:34.725699 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:34.726193 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:34.726200 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:34.726572 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:34.727089 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:34.727103 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:34.728077 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:34.728080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:34.728987 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:34.728998 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:34.729483 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:34.729486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:34.729661 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:34.730260 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:34.730277 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:34.731989 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:34.733812 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:34.733820 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:34.733839 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:34.734198 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:34.734475 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:34.734478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:34.734479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:34.734665 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:34.735076 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:34.735083 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:34.735348 (+ 265us) client_negotiation.cc:769] Sending connection context
0810 03:53:34.735560 (+ 212us) client_negotiation.cc:240] Negotiation successful
0810 03:53:34.735858 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":101,"threads_started":1}
I20260810 03:53:34.738031 23844 heartbeater.cc:347] Connected to a master server at 127.27.153.126:42371
I20260810 03:53:34.738266 23844 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:34.738768 23844 heartbeater.cc:511] Master 127.27.153.126:42371 requested a full tablet report, sending...
I20260810 03:53:34.740371 23513 ts_manager.cc:194] Registered new tserver with Master: fcfc8fa8c7ac40dbb3ee8f78d7270547 (127.27.153.65:38975)
WARNING: no policy specified for kudu/127.27.153.66@KRBTEST.COM; defaulting to no policy
I20260810 03:53:34.741498 23513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:40769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
I20260810 03:53:34.759948 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:42371
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:43087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:53:34.874581 23850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:34.874867 23850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:34.874912 23850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:34.879012 23850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:34.879084 23850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:34.879151 23850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:53:34.884649 23850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:43087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:42371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:34.885850 23850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:34.886947 23850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:34.896950 23865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:34.896950 23868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:34.896960 23864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:34.897212 23850 server_base.cc:1034] running on GCE node
I20260810 03:53:34.897637 23850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:34.898295 23850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:34.899523 23850 hybrid_clock.cc:648] HybridClock initialized: now 1786334014899491 us; error 44 us; skew 500 ppm
Aug 10 03:53:34 dist-test-slave-pww0 krb5kdc[23426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334014, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:34.902513 23850 init.cc:377] Logged in from keytab as kudu/127.27.153.66@KRBTEST.COM (short username kudu)
I20260810 03:53:34.903554 23850 webserver.cc:460] Webserver started at http://127.27.153.66:36955/ using document root <none> and password file <none>
I20260810 03:53:34.904135 23850 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:34.904207 23850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:34.904403 23850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:53:34.906536 23850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "caf97fd4c64e452aaa50b640477abb4a"
format_stamp: "Formatted at 2026-08-10 03:53:34 on dist-test-slave-pww0"
server_key: "60088d7a7f98ff7326cff790a6f3c4c4"
server_key_iv: "483b0cfd0eebae5766cd695cbfe98420"
server_key_version: "encryptionkey@0"
I20260810 03:53:34.907142 23850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "caf97fd4c64e452aaa50b640477abb4a"
format_stamp: "Formatted at 2026-08-10 03:53:34 on dist-test-slave-pww0"
server_key: "60088d7a7f98ff7326cff790a6f3c4c4"
server_key_iv: "483b0cfd0eebae5766cd695cbfe98420"
server_key_version: "encryptionkey@0"
I20260810 03:53:34.911242 23850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:53:34.913895 23879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:34.915189 23850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 03:53:34.915313 23850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "caf97fd4c64e452aaa50b640477abb4a"
format_stamp: "Formatted at 2026-08-10 03:53:34 on dist-test-slave-pww0"
server_key: "60088d7a7f98ff7326cff790a6f3c4c4"
server_key_iv: "483b0cfd0eebae5766cd695cbfe98420"
server_key_version: "encryptionkey@0"
I20260810 03:53:34.915617 23850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:34.929728 23850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:34.932971 23850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:34.933161 23850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:34.933775 23850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:53:34.934615 23850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:53:34.934674 23850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:34.934729 23850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:53:34.934753 23850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:34.943318 23850 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:34091
I20260810 03:53:34.943367 24086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:34091 every 8 connection(s)
I20260810 03:53:34.944295 23850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:53:34.947359 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 23850
I20260810 03:53:34.947527 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:53:34.947819 28261 external_mini_cluster.cc:1428] Setting key 4a22a75055b2d5590ce5ddba8cd9eeee
Aug 10 03:53:34 dist-test-slave-pww0 krb5kdc[23426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334014, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.153.67@KRBTEST.COM; defaulting to no policy
I20260810 03:53:34.958787 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:34.946250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:48863 (local address 127.27.153.126:42371)
0810 03:53:34.946408 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:34.946413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:34.947679 (+ 1266us) server_negotiation.cc:407] Connection header received
0810 03:53:34.948685 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:34.948689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:34.948742 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:34.948819 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:34.950133 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:34.950670 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:34.952230 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:34.952529 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:34.952616 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:34.955211 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:34.955215 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:34.955218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:34.955247 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:34.956597 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:34.957122 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:34.957125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:34.957126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:34.957210 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:34.957551 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:34.957554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:34.957556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:34.957773 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:34.957864 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:34.958460 (+ 596us) server_negotiation.cc:299] Negotiation successful
0810 03:53:34.958601 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:53:34.959717 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:34.946692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.66:48863)
0810 03:53:34.947442 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:34.947534 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:34.948462 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:34.949026 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:34.949035 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:34.949465 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:34.949981 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:34.949995 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:34.950794 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:34.950797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:34.952101 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:34.952115 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:34.952691 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:34.952696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:34.952890 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:34.953483 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:34.953501 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:34.955028 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:34.956720 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:34.956727 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:34.956746 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:34.957021 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:34.957296 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:34.957300 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:34.957302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:34.957458 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:34.957887 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:34.957895 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:34.958178 (+ 283us) client_negotiation.cc:769] Sending connection context
0810 03:53:34.958358 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 03:53:34.958750 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":127,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.67@KRBTEST.COM" created.
I20260810 03:53:34.960898 24088 heartbeater.cc:347] Connected to a master server at 127.27.153.126:42371
I20260810 03:53:34.961143 24088 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:34.961658 24088 heartbeater.cc:511] Master 127.27.153.126:42371 requested a full tablet report, sending...
I20260810 03:53:34.962689 23513 ts_manager.cc:194] Registered new tserver with Master: caf97fd4c64e452aaa50b640477abb4a (127.27.153.66:34091)
I20260810 03:53:34.963248 23513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:48863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
I20260810 03:53:34.975911 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:42371
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:43087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:53:35.090840 24095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:35.091125 24095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:35.091167 24095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:35.095409 24095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:35.095506 24095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:35.095577 24095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:53:35.101140 24095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:43087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:42371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:35.102298 24095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:35.103376 24095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:53:35.113509 24095 server_base.cc:1034] running on GCE node
W20260810 03:53:35.113369 24105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:35.113519 24107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:35.113401 24109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:35.114161 24095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:35.114835 24095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:35.115994 24095 hybrid_clock.cc:648] HybridClock initialized: now 1786334015115974 us; error 32 us; skew 500 ppm
Aug 10 03:53:35 dist-test-slave-pww0 krb5kdc[23426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334015, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:35.118710 24095 init.cc:377] Logged in from keytab as kudu/127.27.153.67@KRBTEST.COM (short username kudu)
I20260810 03:53:35.119684 24095 webserver.cc:460] Webserver started at http://127.27.153.67:43737/ using document root <none> and password file <none>
I20260810 03:53:35.120255 24095 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:35.120332 24095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:35.120539 24095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:53:35.122634 24095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "61990e39cfa44e9ca31271e185bc4787"
format_stamp: "Formatted at 2026-08-10 03:53:35 on dist-test-slave-pww0"
server_key: "e978b61dd2652c3473c9dd890c7b810a"
server_key_iv: "9ce4d17f0e519909a5de42d284425d72"
server_key_version: "encryptionkey@0"
I20260810 03:53:35.123266 24095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "61990e39cfa44e9ca31271e185bc4787"
format_stamp: "Formatted at 2026-08-10 03:53:35 on dist-test-slave-pww0"
server_key: "e978b61dd2652c3473c9dd890c7b810a"
server_key_iv: "9ce4d17f0e519909a5de42d284425d72"
server_key_version: "encryptionkey@0"
I20260810 03:53:35.127722 24095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260810 03:53:35.131093 24120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:35.132512 24095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:53:35.132666 24095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "61990e39cfa44e9ca31271e185bc4787"
format_stamp: "Formatted at 2026-08-10 03:53:35 on dist-test-slave-pww0"
server_key: "e978b61dd2652c3473c9dd890c7b810a"
server_key_iv: "9ce4d17f0e519909a5de42d284425d72"
server_key_version: "encryptionkey@0"
I20260810 03:53:35.132795 24095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:35.147451 24095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:35.151038 24095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:35.151257 24095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:35.152024 24095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:53:35.152963 24095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:53:35.153014 24095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:35.153090 24095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:53:35.153115 24095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:35.163816 24095 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:33387
I20260810 03:53:35.164085 24339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:33387 every 8 connection(s)
I20260810 03:53:35.165194 24095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:53:35.173977 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 24095
I20260810 03:53:35.174129 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:53:35.174441 28261 external_mini_cluster.cc:1428] Setting key c3529c37f84f061e59e3f7a32651ab20
Aug 10 03:53:35 dist-test-slave-pww0 krb5kdc[23426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334015, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:53:35.182706 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:35.167610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:59679 (local address 127.27.153.126:42371)
0810 03:53:35.167764 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:35.167769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:35.168808 (+ 1039us) server_negotiation.cc:407] Connection header received
0810 03:53:35.170459 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:35.170463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:35.170532 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:35.170775 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:35.172387 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.173213 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.174876 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.175173 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.175248 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:35.178877 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:35.178881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:35.178883 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:35.178910 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:35.180517 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:35.181049 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:35.181053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:35.181055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:35.181137 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:35.181524 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:35.181529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:35.181531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:35.181829 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:35.181925 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:35.182458 (+ 533us) server_negotiation.cc:299] Negotiation successful
0810 03:53:35.182583 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 03:53:35.183754 24343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:35.168071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.67:59679)
0810 03:53:35.168642 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:35.168692 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:35.170196 (+ 1504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:35.170960 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:35.170968 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:35.171500 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:35.172183 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:35.172203 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.173350 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.173353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:35.174722 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:35.174735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.175494 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.175498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:35.175662 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:35.177153 (+ 1491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:35.177176 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:35.178668 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:35.180630 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:35.180638 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:35.180657 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:35.180940 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:35.181220 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:35.181223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:35.181225 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:35.181392 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:35.181930 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:35.181935 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:35.182201 (+ 266us) client_negotiation.cc:769] Sending connection context
0810 03:53:35.182359 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 03:53:35.182635 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":138,"threads_started":1}
I20260810 03:53:35.185001 24341 heartbeater.cc:347] Connected to a master server at 127.27.153.126:42371
I20260810 03:53:35.185261 24341 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:35.185849 24341 heartbeater.cc:511] Master 127.27.153.126:42371 requested a full tablet report, sending...
I20260810 03:53:35.186921 23513 ts_manager.cc:194] Registered new tserver with Master: 61990e39cfa44e9ca31271e185bc4787 (127.27.153.67:33387)
I20260810 03:53:35.187608 23513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:59679
I20260810 03:53:35.194224 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:53:35.206403 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:35.196671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55586 (local address 127.27.153.126:42371)
0810 03:53:35.196816 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:35.196822 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:35.197074 (+ 252us) server_negotiation.cc:407] Connection header received
0810 03:53:35.197226 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:35.197229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:35.197290 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:35.197387 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:35.198304 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.199162 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.200414 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.200782 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.200894 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:35.202850 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:35.202854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:35.202857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:35.202910 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:35.204614 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:35.205062 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:35.205066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:35.205068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:35.205164 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:35.205468 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:35.205471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:35.205473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:35.205733 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:35.205862 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:35.206116 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 03:53:35.206240 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:53:35.210064 23513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:53:35.213701 23513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:53:35.229142 24367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:35.224264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43814 (local address 127.27.153.65:38975)
0810 03:53:35.224634 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:35.224648 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:35.224665 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:53:35.224752 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:35.224756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:35.224898 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:35.225034 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:53:35.225419 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.226494 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.227805 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.228450 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.228553 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:35.228729 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:35.228879 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 03:53:35.228980 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":98,"threads_started":1}
I20260810 03:53:35.229128 24365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:35.223723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:38975 (local address 127.0.0.1:43814)
0810 03:53:35.223944 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:35.223957 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:35.224040 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:35.225190 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:35.225193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:35.225209 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:53:35.225296 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:35.225302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.226616 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.226619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:35.227657 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:35.227666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.228571 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.228575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:35.228691 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:35.228732 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 03:53:35.228840 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 03:53:35.228974 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":90,"threads_started":1}
I20260810 03:53:35.229413 24362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:35.223528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:33387 (local address 127.0.0.1:39528)
0810 03:53:35.223819 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:35.223831 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:35.223921 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:35.224672 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:35.224690 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:35.224713 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:53:35.224877 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:35.224885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.226449 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.226452 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:35.227864 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:35.227876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.228998 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.229002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:35.229128 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:35.229149 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 03:53:35.229220 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 03:53:35.229291 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":85,"thread_start_us":117,"threads_started":1}
I20260810 03:53:35.230042 24356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:35.222990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:34091 (local address 127.0.0.1:51502)
0810 03:53:35.223357 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:35.223414 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:35.223584 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:35.225011 (+ 1427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:35.225014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:35.225032 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:53:35.225137 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:35.225143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.226312 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.226315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:35.227864 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:35.227876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.228998 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.229002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:35.229753 (+ 751us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:35.229777 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:53:35.229849 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 03:53:35.229917 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":686,"thread_start_us":80,"threads_started":1}
I20260810 03:53:35.230124 24361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:35.223097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51502 (local address 127.27.153.66:34091)
0810 03:53:35.224545 (+ 1448us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:35.224557 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:35.224581 (+ 24us) server_negotiation.cc:407] Connection header received
0810 03:53:35.224658 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:35.224664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:35.224850 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:35.225300 (+ 450us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:53:35.225470 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.226182 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.228010 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.228857 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.229111 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:35.229260 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:35.229888 (+ 628us) server_negotiation.cc:299] Negotiation successful
0810 03:53:35.229992 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1339,"thread_start_us":125,"threads_started":1}
I20260810 03:53:35.230779 24366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:35.223825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39528 (local address 127.27.153.67:33387)
0810 03:53:35.224195 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:35.224205 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:35.224226 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:53:35.224290 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:35.224294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:35.224498 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:35.225036 (+ 538us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:53:35.225196 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.226319 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.228011 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.228857 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.230340 (+ 1483us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:35.230494 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:35.230542 (+ 48us) server_negotiation.cc:299] Negotiation successful
0810 03:53:35.230638 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":95,"threads_started":1}
I20260810 03:53:35.232689 23965 tablet_service.cc:1467] Processing CreateTablet for tablet 12eaf68c75a94e2f961e27f37a03f059 (DEFAULT_TABLE table=test-table [id=0359e53ee72a4f7aa90091466178126e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:53:35.232517 23732 tablet_service.cc:1467] Processing CreateTablet for tablet 12eaf68c75a94e2f961e27f37a03f059 (DEFAULT_TABLE table=test-table [id=0359e53ee72a4f7aa90091466178126e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:53:35.233624 23965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12eaf68c75a94e2f961e27f37a03f059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:53:35.233667 23732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12eaf68c75a94e2f961e27f37a03f059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:53:35.233394 24215 tablet_service.cc:1467] Processing CreateTablet for tablet 12eaf68c75a94e2f961e27f37a03f059 (DEFAULT_TABLE table=test-table [id=0359e53ee72a4f7aa90091466178126e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:53:35.234378 24215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12eaf68c75a94e2f961e27f37a03f059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:53:35.240175 24371 tablet_bootstrap.cc:492] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a: Bootstrap starting.
I20260810 03:53:35.242111 24373 tablet_bootstrap.cc:492] T 12eaf68c75a94e2f961e27f37a03f059 P 61990e39cfa44e9ca31271e185bc4787: Bootstrap starting.
I20260810 03:53:35.243027 24372 tablet_bootstrap.cc:492] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Bootstrap starting.
I20260810 03:53:35.243183 24371 tablet_bootstrap.cc:654] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a: Neither blocks nor log segments found. Creating new log.
I20260810 03:53:35.243990 24371 log.cc:824] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:35.244418 24373 tablet_bootstrap.cc:654] T 12eaf68c75a94e2f961e27f37a03f059 P 61990e39cfa44e9ca31271e185bc4787: Neither blocks nor log segments found. Creating new log.
I20260810 03:53:35.245203 24372 tablet_bootstrap.cc:654] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Neither blocks nor log segments found. Creating new log.
I20260810 03:53:35.245669 24373 log.cc:824] T 12eaf68c75a94e2f961e27f37a03f059 P 61990e39cfa44e9ca31271e185bc4787: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:35.245919 24371 tablet_bootstrap.cc:492] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a: No bootstrap required, opened a new log
I20260810 03:53:35.246007 24372 log.cc:824] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:35.246126 24371 ts_tablet_manager.cc:1397] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:53:35.247723 24373 tablet_bootstrap.cc:492] T 12eaf68c75a94e2f961e27f37a03f059 P 61990e39cfa44e9ca31271e185bc4787: No bootstrap required, opened a new log
I20260810 03:53:35.247920 24373 ts_tablet_manager.cc:1397] T 12eaf68c75a94e2f961e27f37a03f059 P 61990e39cfa44e9ca31271e185bc4787: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:53:35.248780 24372 tablet_bootstrap.cc:492] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547: No bootstrap required, opened a new log
I20260810 03:53:35.248973 24372 ts_tablet_manager.cc:1397] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:53:35.249573 24371 raft_consensus.cc:348] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } }
I20260810 03:53:35.249816 24371 raft_consensus.cc:374] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:53:35.249871 24371 raft_consensus.cc:729] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caf97fd4c64e452aaa50b640477abb4a, State: Initialized, Role: FOLLOWER
I20260810 03:53:35.250322 24371 consensus_queue.cc:260] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } }
I20260810 03:53:35.251029 24088 heartbeater.cc:503] Master 127.27.153.126:42371 was elected leader, sending a full tablet report...
I20260810 03:53:35.251296 24371 ts_tablet_manager.cc:1428] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 03:53:35.252527 24373 raft_consensus.cc:348] T 12eaf68c75a94e2f961e27f37a03f059 P 61990e39cfa44e9ca31271e185bc4787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } }
I20260810 03:53:35.252828 24373 raft_consensus.cc:374] T 12eaf68c75a94e2f961e27f37a03f059 P 61990e39cfa44e9ca31271e185bc4787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:53:35.252897 24373 raft_consensus.cc:729] T 12eaf68c75a94e2f961e27f37a03f059 P 61990e39cfa44e9ca31271e185bc4787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61990e39cfa44e9ca31271e185bc4787, State: Initialized, Role: FOLLOWER
I20260810 03:53:35.253388 24373 consensus_queue.cc:260] T 12eaf68c75a94e2f961e27f37a03f059 P 61990e39cfa44e9ca31271e185bc4787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } }
I20260810 03:53:35.253443 24372 raft_consensus.cc:348] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } }
I20260810 03:53:35.253679 24372 raft_consensus.cc:374] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:53:35.253741 24372 raft_consensus.cc:729] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcfc8fa8c7ac40dbb3ee8f78d7270547, State: Initialized, Role: FOLLOWER
I20260810 03:53:35.254138 24372 consensus_queue.cc:260] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } }
I20260810 03:53:35.254505 24373 ts_tablet_manager.cc:1428] T 12eaf68c75a94e2f961e27f37a03f059 P 61990e39cfa44e9ca31271e185bc4787: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260810 03:53:35.254858 24341 heartbeater.cc:503] Master 127.27.153.126:42371 was elected leader, sending a full tablet report...
I20260810 03:53:35.256058 24372 ts_tablet_manager.cc:1428] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 03:53:35.256398 23844 heartbeater.cc:503] Master 127.27.153.126:42371 was elected leader, sending a full tablet report...
I20260810 03:53:35.388994 24380 raft_consensus.cc:482] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:53:35.389230 24380 raft_consensus.cc:504] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } }
I20260810 03:53:35.390465 24380 leader_election.cc:288] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcfc8fa8c7ac40dbb3ee8f78d7270547 (127.27.153.65:38975), 61990e39cfa44e9ca31271e185bc4787 (127.27.153.67:33387)
I20260810 03:53:35.394075 24366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:35.390895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:41509 (local address 127.27.153.67:33387)
0810 03:53:35.391017 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:35.391022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:35.391040 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:53:35.391129 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:35.391132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:35.391174 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:35.391270 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:53:35.391571 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.392081 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.393094 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.393502 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.393612 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:35.393670 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:35.393876 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 03:53:35.393951 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 03:53:35.394075 24091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:35.390787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:33387 (local address 127.27.153.66:41509)
0810 03:53:35.390922 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:35.390938 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:35.391035 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:35.391295 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:35.391298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:35.391320 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:53:35.391433 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:35.391439 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.392186 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.392189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:35.392972 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:35.392981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.393613 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.393616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:35.393760 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:35.393782 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 03:53:35.393846 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 03:53:35.393922 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 03:53:35.394579 24367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:35.390893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:38537 (local address 127.27.153.65:38975)
0810 03:53:35.391017 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:35.391022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:35.391584 (+ 562us) server_negotiation.cc:407] Connection header received
0810 03:53:35.391690 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:35.391692 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:35.391744 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:35.391807 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:53:35.392056 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.392554 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.393544 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.393976 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.394089 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:35.394140 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:35.394380 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 03:53:35.394462 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 03:53:35.394975 24384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:35.390787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:38975 (local address 127.27.153.66:38537)
0810 03:53:35.391484 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:35.391499 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:35.391603 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:35.391822 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:35.391825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:35.391841 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:53:35.391953 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:35.391959 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.392656 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.392660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:35.393424 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:35.393433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.394122 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.394126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:35.394220 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:35.394258 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 03:53:35.394733 (+ 475us) client_negotiation.cc:240] Negotiation successful
0810 03:53:35.394837 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":313,"thread_start_us":135,"threads_started":1}
I20260810 03:53:35.394896 24250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12eaf68c75a94e2f961e27f37a03f059" candidate_uuid: "caf97fd4c64e452aaa50b640477abb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61990e39cfa44e9ca31271e185bc4787" is_pre_election: true
I20260810 03:53:35.395272 24250 raft_consensus.cc:2454] T 12eaf68c75a94e2f961e27f37a03f059 P 61990e39cfa44e9ca31271e185bc4787 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caf97fd4c64e452aaa50b640477abb4a in term 0.
I20260810 03:53:35.395669 23763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12eaf68c75a94e2f961e27f37a03f059" candidate_uuid: "caf97fd4c64e452aaa50b640477abb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" is_pre_election: true
I20260810 03:53:35.395877 23884 leader_election.cc:302] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61990e39cfa44e9ca31271e185bc4787, caf97fd4c64e452aaa50b640477abb4a; no voters:
I20260810 03:53:35.395959 23763 raft_consensus.cc:2454] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caf97fd4c64e452aaa50b640477abb4a in term 0.
I20260810 03:53:35.396158 24380 raft_consensus.cc:2781] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:53:35.396256 24380 raft_consensus.cc:482] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:53:35.396311 24380 raft_consensus.cc:3037] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:53:35.397406 24380 raft_consensus.cc:504] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } }
I20260810 03:53:35.397744 24380 leader_election.cc:288] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [CANDIDATE]: Term 1 election: Requested vote from peers fcfc8fa8c7ac40dbb3ee8f78d7270547 (127.27.153.65:38975), 61990e39cfa44e9ca31271e185bc4787 (127.27.153.67:33387)
I20260810 03:53:35.398129 23763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12eaf68c75a94e2f961e27f37a03f059" candidate_uuid: "caf97fd4c64e452aaa50b640477abb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547"
I20260810 03:53:35.398239 23763 raft_consensus.cc:3037] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:53:35.398250 24250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12eaf68c75a94e2f961e27f37a03f059" candidate_uuid: "caf97fd4c64e452aaa50b640477abb4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61990e39cfa44e9ca31271e185bc4787"
I20260810 03:53:35.398399 24250 raft_consensus.cc:3037] T 12eaf68c75a94e2f961e27f37a03f059 P 61990e39cfa44e9ca31271e185bc4787 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:53:35.399259 23763 raft_consensus.cc:2454] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caf97fd4c64e452aaa50b640477abb4a in term 1.
I20260810 03:53:35.399514 24250 raft_consensus.cc:2454] T 12eaf68c75a94e2f961e27f37a03f059 P 61990e39cfa44e9ca31271e185bc4787 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caf97fd4c64e452aaa50b640477abb4a in term 1.
I20260810 03:53:35.399623 23880 leader_election.cc:302] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: caf97fd4c64e452aaa50b640477abb4a, fcfc8fa8c7ac40dbb3ee8f78d7270547; no voters:
I20260810 03:53:35.399839 24380 raft_consensus.cc:2781] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:53:35.400068 24380 raft_consensus.cc:686] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [term 1 LEADER]: Becoming Leader. State: Replica: caf97fd4c64e452aaa50b640477abb4a, State: Running, Role: LEADER
I20260810 03:53:35.400370 24380 consensus_queue.cc:237] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } }
I20260810 03:53:35.403847 23513 catalog_manager.cc:5495] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a reported cstate change: term changed from 0 to 1, leader changed from <none> to caf97fd4c64e452aaa50b640477abb4a (127.27.153.66). New cstate: current_term: 1 leader_uuid: "caf97fd4c64e452aaa50b640477abb4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } health_report { overall_health: UNKNOWN } } }
W20260810 03:53:35.416429 24342 tablet.cc:2369] T 12eaf68c75a94e2f961e27f37a03f059 P 61990e39cfa44e9ca31271e185bc4787: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:53:35.422814 24361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:35.419080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51510 (local address 127.27.153.66:34091)
0810 03:53:35.419207 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:35.419211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:35.419225 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:53:35.419371 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:35.419373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:35.419432 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:35.419716 (+ 284us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:53:35.419953 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.420459 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.421321 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:35.421597 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:35.421685 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:35.421990 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:53:35.422450 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:53:35.422543 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:35.422588 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 03:53:35.422648 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 03:53:35.429116 24250 raft_consensus.cc:1261] T 12eaf68c75a94e2f961e27f37a03f059 P 61990e39cfa44e9ca31271e185bc4787 [term 1 FOLLOWER]: Refusing update from remote peer caf97fd4c64e452aaa50b640477abb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:53:35.429150 23763 raft_consensus.cc:1261] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Refusing update from remote peer caf97fd4c64e452aaa50b640477abb4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:53:35.429946 24385 consensus_queue.cc:1035] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:53:35.430145 24380 consensus_queue.cc:1035] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:53:35.440963 24393 mvcc.cc:205] Tried to move back new op lower bound from 7316824127189557248 to 7316824127082512384. Current Snapshot: MvccSnapshot[applied={T|T < 7316824127189557248}]
I20260810 03:53:35.449788 23513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:53:35.449999 23513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:53:35.453052 23513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be: Sending DeleteTablet for 3 replicas of tablet 12eaf68c75a94e2f961e27f37a03f059
I20260810 03:53:35.454051 24215 tablet_service.cc:1514] Processing DeleteTablet for tablet 12eaf68c75a94e2f961e27f37a03f059 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:53:35 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:39528
I20260810 03:53:35.454070 23732 tablet_service.cc:1514] Processing DeleteTablet for tablet 12eaf68c75a94e2f961e27f37a03f059 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:53:35 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:43814
I20260810 03:53:35.454223 23965 tablet_service.cc:1514] Processing DeleteTablet for tablet 12eaf68c75a94e2f961e27f37a03f059 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:53:35 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:51502
I20260810 03:53:35.454721 24418 tablet_replica.cc:330] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547: stopping tablet replica
I20260810 03:53:35.454903 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 23618
I20260810 03:53:35.455114 24418 raft_consensus.cc:2229] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:53:35.455636 24419 tablet_replica.cc:330] T 12eaf68c75a94e2f961e27f37a03f059 P 61990e39cfa44e9ca31271e185bc4787: stopping tablet replica
I20260810 03:53:35.456027 24419 raft_consensus.cc:2229] T 12eaf68c75a94e2f961e27f37a03f059 P 61990e39cfa44e9ca31271e185bc4787 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:53:35.456418 24419 raft_consensus.cc:2258] T 12eaf68c75a94e2f961e27f37a03f059 P 61990e39cfa44e9ca31271e185bc4787 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:53:35.480170 24419 ts_tablet_manager.cc:1905] T 12eaf68c75a94e2f961e27f37a03f059 P 61990e39cfa44e9ca31271e185bc4787: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:53:35.482690 24419 ts_tablet_manager.cc:1918] T 12eaf68c75a94e2f961e27f37a03f059 P 61990e39cfa44e9ca31271e185bc4787: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:53:35.482821 24419 log.cc:1196] T 12eaf68c75a94e2f961e27f37a03f059 P 61990e39cfa44e9ca31271e185bc4787: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/12eaf68c75a94e2f961e27f37a03f059
I20260810 03:53:35.483177 24419 ts_tablet_manager.cc:1939] T 12eaf68c75a94e2f961e27f37a03f059 P 61990e39cfa44e9ca31271e185bc4787: Deleting consensus metadata
I20260810 03:53:35.483359 24420 tablet_replica.cc:330] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a: stopping tablet replica
I20260810 03:53:35.483855 24420 raft_consensus.cc:2229] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:53:35.484334 24420 raft_consensus.cc:2258] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:53:35.484733 23488 catalog_manager.cc:4841] TS 61990e39cfa44e9ca31271e185bc4787 (127.27.153.67:33387): tablet 12eaf68c75a94e2f961e27f37a03f059 (table test-table [id=0359e53ee72a4f7aa90091466178126e]) successfully deleted
W20260810 03:53:35.486487 23483 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.153.65:38975 (error 108)
I20260810 03:53:35.487120 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 23850
I20260810 03:53:35.487447 24420 ts_tablet_manager.cc:1905] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:53:35.487622 24356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:35.487235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:38975 (local address 127.0.0.1:43828)
0810 03:53:35.487401 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:35.487527 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:38975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 03:53:35.487815 23483 catalog_manager.cc:4584] TS fcfc8fa8c7ac40dbb3ee8f78d7270547 (127.27.153.65:38975): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12eaf68c75a94e2f961e27f37a03f059: Network error: Client connection negotiation failed: client connection to 127.27.153.65:38975: connect: Connection refused (error 111)
I20260810 03:53:35.520658 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 24095
I20260810 03:53:35.524047 24356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:35.523689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:34091 (local address 127.0.0.1:51522)
0810 03:53:35.523901 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:35.523952 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.66:34091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260810 03:53:35.535514 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 23457
I20260810 03:53:55.556252 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:42371
--webserver_interface=127.27.153.126
--webserver_port=41403
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:43087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:42371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:53:55.674786 24422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:55.675089 24422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:55.675150 24422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:55.679250 24422 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 03:53:55.679334 24422 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 03:53:55.679356 24422 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 03:53:55.679373 24422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:53:55.679396 24422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:55.679419 24422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:55.679447 24422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:53:55.679499 24422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:53:55.685271 24422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:43087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:42371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:42371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=41403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:55.686425 24422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:55.687518 24422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:55.702327 24440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:55.702330 24435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:55.702329 24434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:55.702407 24422 server_base.cc:1034] running on GCE node
I20260810 03:53:55.703291 24422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:55.704618 24422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:55.705798 24422 hybrid_clock.cc:648] HybridClock initialized: now 1786334035705773 us; error 45 us; skew 500 ppm
Aug 10 03:53:55 dist-test-slave-pww0 krb5kdc[23426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334035, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:55.708796 24422 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:53:55.709842 24422 webserver.cc:460] Webserver started at http://127.27.153.126:41403/ using document root <none> and password file <none>
I20260810 03:53:55.710455 24422 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:55.710551 24422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:55.715044 24422 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:53:55.717571 24452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:55.719002 24422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 03:53:55.719156 24422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "356ae6557eff400d8b1de99f512af1be"
format_stamp: "Formatted at 2026-08-10 03:53:34 on dist-test-slave-pww0"
server_key: "ff91e49bdb94950e2014f72683062e60"
server_key_iv: "cc8f5d442a8dbc83e604b1f755b7b470"
server_key_version: "encryptionkey@0"
I20260810 03:53:55.719848 24422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:55.750663 24422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:55.754195 24422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:55.754402 24422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:55.761916 24422 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:42371
I20260810 03:53:55.762181 24541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:42371 every 8 connection(s)
I20260810 03:53:55.763092 24422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:53:55.765406 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 24422
I20260810 03:53:55.765882 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:38975
--local_ip_for_outbound_sockets=127.27.153.65
--tserver_master_addrs=127.27.153.126:42371
--webserver_port=38379
--webserver_interface=127.27.153.65
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:43087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:53:55.768394 24544 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:53:55.770191 24544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be: Bootstrap starting.
I20260810 03:53:55.788142 24544 log.cc:824] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:55.797474 24544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:53:55.797865 24544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be: Bootstrap complete.
I20260810 03:53:55.802106 24544 raft_consensus.cc:348] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356ae6557eff400d8b1de99f512af1be" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42371 } }
I20260810 03:53:55.802559 24544 raft_consensus.cc:729] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 356ae6557eff400d8b1de99f512af1be, State: Initialized, Role: FOLLOWER
I20260810 03:53:55.803061 24544 consensus_queue.cc:260] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356ae6557eff400d8b1de99f512af1be" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42371 } }
I20260810 03:53:55.803184 24544 raft_consensus.cc:388] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:53:55.803257 24544 raft_consensus.cc:482] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:53:55.803357 24544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 1 FOLLOWER]: Advancing to term 2
I20260810 03:53:55.805442 24544 raft_consensus.cc:504] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356ae6557eff400d8b1de99f512af1be" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42371 } }
I20260810 03:53:55.805797 24544 leader_election.cc:302] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 356ae6557eff400d8b1de99f512af1be; no voters:
I20260810 03:53:55.806123 24544 leader_election.cc:288] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 03:53:55.806234 24548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 2 FOLLOWER]: Leader election won for term 2
I20260810 03:53:55.806442 24548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 2 LEADER]: Becoming Leader. State: Replica: 356ae6557eff400d8b1de99f512af1be, State: Running, Role: LEADER
I20260810 03:53:55.806766 24548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356ae6557eff400d8b1de99f512af1be" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42371 } }
I20260810 03:53:55.807509 24544 sys_catalog.cc:564] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:53:55.808234 24549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "356ae6557eff400d8b1de99f512af1be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356ae6557eff400d8b1de99f512af1be" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42371 } } }
I20260810 03:53:55.808373 24549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [sys.catalog]: This master's current role is: LEADER
I20260810 03:53:55.809029 24550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 356ae6557eff400d8b1de99f512af1be. Latest consensus state: current_term: 2 leader_uuid: "356ae6557eff400d8b1de99f512af1be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356ae6557eff400d8b1de99f512af1be" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42371 } } }
I20260810 03:53:55.809206 24550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [sys.catalog]: This master's current role is: LEADER
I20260810 03:53:55.809451 24561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:53:55.814857 24561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:53:55.815567 24561 catalog_manager.cc:1259] Loaded cluster ID: b77c67349c6342e4932dc89ba6409c7d
I20260810 03:53:55.815634 24561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:53:55.818154 24561 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:53:55.818933 24561 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be: Loaded TSK: 0
I20260810 03:53:55.836594 24561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be: Generated new TSK 1
I20260810 03:53:55.836922 24561 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 03:53:55.891217 24546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:55.891554 24546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:55.891621 24546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:55.895871 24546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:55.895949 24546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:55.896036 24546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:53:55.901918 24546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:43087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:38975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=38379
--tserver_master_addrs=127.27.153.126:42371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:55.903080 24546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:55.904222 24546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:55.914572 24592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:55.914572 24596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:55.914635 24593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:55.915030 24546 server_base.cc:1034] running on GCE node
I20260810 03:53:55.915608 24546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:55.916504 24546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:55.917721 24546 hybrid_clock.cc:648] HybridClock initialized: now 1786334035917685 us; error 50 us; skew 500 ppm
Aug 10 03:53:55 dist-test-slave-pww0 krb5kdc[23426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334035, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:55.921629 24546 init.cc:377] Logged in from keytab as kudu/127.27.153.65@KRBTEST.COM (short username kudu)
I20260810 03:53:55.923087 24546 webserver.cc:460] Webserver started at http://127.27.153.65:38379/ using document root <none> and password file <none>
I20260810 03:53:55.924027 24546 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:55.924155 24546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:55.929899 24546 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 03:53:55.932950 24605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:55.934541 24546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:53:55.934682 24546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547"
format_stamp: "Formatted at 2026-08-10 03:53:34 on dist-test-slave-pww0"
server_key: "ffc2269d8c50241c17c7978e2a131963"
server_key_iv: "91b3579162793ec72b95edd75cadfc68"
server_key_version: "encryptionkey@0"
I20260810 03:53:55.935436 24546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:55.961824 24546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:55.965417 24546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:55.965626 24546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:55.966264 24546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:53:55.967486 24619 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 03:53:55.971810 24546 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 03:53:55.971901 24546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260810 03:53:55.971972 24546 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 03:53:55.973838 24546 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 03:53:55.973892 24546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 03:53:55.974263 24619 tablet_bootstrap.cc:492] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Bootstrap starting.
I20260810 03:53:55.984625 24546 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:38975
I20260810 03:53:55.984774 24813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:38975 every 8 connection(s)
I20260810 03:53:55.985761 24546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:53:55.994002 24619 log.cc:824] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:55.994676 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 24546
I20260810 03:53:55.995189 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:34091
--local_ip_for_outbound_sockets=127.27.153.66
--tserver_master_addrs=127.27.153.126:42371
--webserver_port=36955
--webserver_interface=127.27.153.66
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:43087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 03:53:55 dist-test-slave-pww0 krb5kdc[23426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334035, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:53:56.004513 24619 tablet_bootstrap.cc:492] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:53:56.004791 24820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:55.988162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.65:33995)
0810 03:53:55.988842 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:55.988910 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:55.990060 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:55.991390 (+ 1330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:55.991399 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:55.992031 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:55.992723 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:55.992745 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:55.994222 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:55.994226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:55.995374 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:55.995386 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:55.996670 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:55.996675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:55.996867 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:55.997550 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:55.997573 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:55.999351 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:56.001535 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:56.001541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:56.001555 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:56.001812 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:56.002148 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:56.002151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:56.002153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:56.002286 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:56.002868 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:56.002874 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:56.003123 (+ 249us) client_negotiation.cc:769] Sending connection context
0810 03:53:56.003286 (+ 163us) client_negotiation.cc:240] Negotiation successful
0810 03:53:56.003670 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":145,"threads_started":1}
I20260810 03:53:56.004998 24619 tablet_bootstrap.cc:492] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Bootstrap complete.
I20260810 03:53:56.004830 24818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:55.988190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:33995 (local address 127.27.153.126:42371)
0810 03:53:55.988623 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:55.988647 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:55.989072 (+ 425us) server_negotiation.cc:407] Connection header received
0810 03:53:55.990595 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:55.990608 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:55.990936 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:55.991245 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:55.992948 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:55.994032 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:55.995550 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:55.996277 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:55.996452 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:55.999685 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:55.999692 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:55.999707 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:55.999746 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:56.001413 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:56.001926 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:56.001931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:56.001935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:56.002056 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:56.002376 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:56.002379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:56.002380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:56.002755 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:56.002879 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:56.003421 (+ 542us) server_negotiation.cc:299] Negotiation successful
0810 03:53:56.003737 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":127,"threads_started":1}
I20260810 03:53:56.006445 24619 ts_tablet_manager.cc:1397] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.011s
I20260810 03:53:56.007818 24815 heartbeater.cc:347] Connected to a master server at 127.27.153.126:42371
I20260810 03:53:56.008068 24815 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:56.008589 24815 heartbeater.cc:511] Master 127.27.153.126:42371 requested a full tablet report, sending...
I20260810 03:53:56.009718 24619 raft_consensus.cc:348] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } }
I20260810 03:53:56.010313 24619 raft_consensus.cc:729] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcfc8fa8c7ac40dbb3ee8f78d7270547, State: Initialized, Role: FOLLOWER
I20260810 03:53:56.010859 24619 consensus_queue.cc:260] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } }
I20260810 03:53:56.011255 24481 ts_manager.cc:194] Registered new tserver with Master: fcfc8fa8c7ac40dbb3ee8f78d7270547 (127.27.153.65:38975)
I20260810 03:53:56.011971 24619 ts_tablet_manager.cc:1428] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Time spent starting tablet: real 0.005s user 0.008s sys 0.000s
I20260810 03:53:56.014936 24481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:33995
I20260810 03:53:56.017750 24815 heartbeater.cc:503] Master 127.27.153.126:42371 was elected leader, sending a full tablet report...
Aug 10 03:53:56 dist-test-slave-pww0 krb5kdc[23426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334035, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for kudu/127.27.153.65@KRBTEST.COM
I20260810 03:53:56.026177 24826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:56.014367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:38975 (local address 127.0.0.1:56638)
0810 03:53:56.015034 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:56.015090 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:56.015239 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:56.015812 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:56.015818 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:56.016301 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:56.016464 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:56.016474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.017790 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.017794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:56.018823 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:56.018835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.019756 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.019759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:56.019876 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:56.020434 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:56.020449 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:56.022060 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:56.024221 (+ 2161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:56.024230 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:56.024236 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:56.024548 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:56.024915 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:56.024917 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:56.024919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:56.025002 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:56.025491 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:56.025497 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:56.025618 (+ 121us) client_negotiation.cc:769] Sending connection context
0810 03:53:56.025742 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 03:53:56.025962 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":140,"threads_started":1}
I20260810 03:53:56.026177 24827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:56.014470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56638 (local address 127.27.153.65:38975)
0810 03:53:56.015269 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:56.015281 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:56.015340 (+ 59us) server_negotiation.cc:407] Connection header received
0810 03:53:56.015418 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:56.015422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:56.015662 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:56.015812 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:56.016612 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.017666 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.018960 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.019632 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.019729 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:56.022203 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:56.022213 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:56.022224 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:56.022249 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:56.024090 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:56.024716 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:56.024721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:56.024722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:56.024811 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:56.025087 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:56.025091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:56.025093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:56.025364 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:56.025481 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:56.025841 (+ 360us) server_negotiation.cc:299] Negotiation successful
0810 03:53:56.026009 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":127,"threads_started":1}
I20260810 03:53:56.027280 24698 tablet_service.cc:1514] Processing DeleteTablet for tablet 12eaf68c75a94e2f961e27f37a03f059 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:53:35 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:56638
I20260810 03:53:56.028367 24828 tablet_replica.cc:330] stopping tablet replica
I20260810 03:53:56.028672 24828 raft_consensus.cc:2229] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:53:56.028908 24828 raft_consensus.cc:2258] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:53:56.029909 24828 ts_tablet_manager.cc:1905] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:53:56.033648 24828 ts_tablet_manager.cc:1918] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:53:56.033752 24828 log.cc:1196] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/wal/wals/12eaf68c75a94e2f961e27f37a03f059
I20260810 03:53:56.033987 24828 ts_tablet_manager.cc:1939] T 12eaf68c75a94e2f961e27f37a03f059 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Deleting consensus metadata
I20260810 03:53:56.035851 24455 catalog_manager.cc:4841] TS fcfc8fa8c7ac40dbb3ee8f78d7270547 (127.27.153.65:38975): tablet 12eaf68c75a94e2f961e27f37a03f059 (table test-table [id=0359e53ee72a4f7aa90091466178126e]) successfully deleted
W20260810 03:53:56.119053 24822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:56.119354 24822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:56.119400 24822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:56.123528 24822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:56.123615 24822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:56.123685 24822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:53:56.129382 24822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:43087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:34091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=36955
--tserver_master_addrs=127.27.153.126:42371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:56.130558 24822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:56.131611 24822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:56.141696 24841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:56.141698 24845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:56.141834 24822 server_base.cc:1034] running on GCE node
W20260810 03:53:56.141865 24850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:56.142421 24822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:56.143139 24822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:56.144337 24822 hybrid_clock.cc:648] HybridClock initialized: now 1786334036144316 us; error 35 us; skew 500 ppm
Aug 10 03:53:56 dist-test-slave-pww0 krb5kdc[23426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334036, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:56.147246 24822 init.cc:377] Logged in from keytab as kudu/127.27.153.66@KRBTEST.COM (short username kudu)
I20260810 03:53:56.148334 24822 webserver.cc:460] Webserver started at http://127.27.153.66:36955/ using document root <none> and password file <none>
I20260810 03:53:56.148939 24822 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:56.149034 24822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:56.153376 24822 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:53:56.155961 24860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:56.157425 24822 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:53:56.157577 24822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "caf97fd4c64e452aaa50b640477abb4a"
format_stamp: "Formatted at 2026-08-10 03:53:34 on dist-test-slave-pww0"
server_key: "60088d7a7f98ff7326cff790a6f3c4c4"
server_key_iv: "483b0cfd0eebae5766cd695cbfe98420"
server_key_version: "encryptionkey@0"
I20260810 03:53:56.158177 24822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:56.186175 24822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:56.189415 24822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:56.189612 24822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:56.190245 24822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:53:56.191397 24870 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 03:53:56.196050 24822 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 03:53:56.196142 24822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260810 03:53:56.196218 24822 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 03:53:56.198032 24822 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 03:53:56.198084 24822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 03:53:56.198385 24870 tablet_bootstrap.cc:492] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a: Bootstrap starting.
I20260810 03:53:56.204352 24870 log.cc:824] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:56.208390 24822 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:34091
I20260810 03:53:56.208525 25071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:34091 every 8 connection(s)
I20260810 03:53:56.209364 24822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:53:56.213891 24870 tablet_bootstrap.cc:492] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:53:56.214293 24870 tablet_bootstrap.cc:492] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a: Bootstrap complete.
I20260810 03:53:56.214656 24870 ts_tablet_manager.cc:1397] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20260810 03:53:56.215265 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 24822
I20260810 03:53:56.215817 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:33387
--local_ip_for_outbound_sockets=127.27.153.67
--tserver_master_addrs=127.27.153.126:42371
--webserver_port=43737
--webserver_interface=127.27.153.67
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:43087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:53:56.217383 24870 raft_consensus.cc:348] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } }
I20260810 03:53:56.217859 24870 raft_consensus.cc:729] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: caf97fd4c64e452aaa50b640477abb4a, State: Initialized, Role: FOLLOWER
I20260810 03:53:56.218410 24870 consensus_queue.cc:260] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } }
I20260810 03:53:56.219548 24870 ts_tablet_manager.cc:1428] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
Aug 10 03:53:56 dist-test-slave-pww0 krb5kdc[23426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334036, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:53:56.224874 24818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:56.211412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:50757 (local address 127.27.153.126:42371)
0810 03:53:56.211586 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:56.211591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:56.212229 (+ 638us) server_negotiation.cc:407] Connection header received
0810 03:53:56.213229 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:56.213233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:56.213298 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:56.213403 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:56.214641 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.215443 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.217034 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.217371 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.217453 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:56.220766 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:56.220769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:56.220772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:56.220801 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:56.222292 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:56.223022 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:56.223027 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:56.223029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:56.223138 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:56.223549 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:56.223553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:56.223555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:56.223828 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:56.223942 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:56.224512 (+ 570us) server_negotiation.cc:299] Negotiation successful
0810 03:53:56.224691 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 03:53:56.225656 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:56.211564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.66:50757)
0810 03:53:56.212062 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:56.212116 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:56.213013 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:56.213551 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:56.213558 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:56.213946 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:56.214479 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:56.214497 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.215640 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.215643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:56.216881 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:56.216893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.217629 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.217632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:56.217859 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:56.218838 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:56.218856 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:56.220560 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:56.222427 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:56.222434 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:56.222451 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:56.222851 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:56.223271 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:56.223274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:56.223275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:56.223429 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:56.223932 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:56.223938 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:56.224251 (+ 313us) client_negotiation.cc:769] Sending connection context
0810 03:53:56.224412 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 03:53:56.224713 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":134,"threads_started":1}
I20260810 03:53:56.226868 25072 heartbeater.cc:347] Connected to a master server at 127.27.153.126:42371
I20260810 03:53:56.227105 25072 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:56.227674 25072 heartbeater.cc:511] Master 127.27.153.126:42371 requested a full tablet report, sending...
I20260810 03:53:56.229444 24481 ts_manager.cc:194] Registered new tserver with Master: caf97fd4c64e452aaa50b640477abb4a (127.27.153.66:34091)
I20260810 03:53:56.230829 24481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:50757
I20260810 03:53:56.233011 25072 heartbeater.cc:503] Master 127.27.153.126:42371 was elected leader, sending a full tablet report...
Aug 10 03:53:56 dist-test-slave-pww0 krb5kdc[23426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334035, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for kudu/127.27.153.66@KRBTEST.COM
I20260810 03:53:56.242695 24826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:56.230507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:34091 (local address 127.0.0.1:43634)
0810 03:53:56.230767 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:56.230784 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:56.230873 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:56.233455 (+ 2582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:56.233458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:56.233671 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:56.233785 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:56.233793 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.235271 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.235274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:56.235923 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:56.235934 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.236785 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.236788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:56.236869 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:56.237413 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:56.237426 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:56.238890 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:56.241063 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:56.241066 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:56.241067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:56.241316 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:56.241639 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:56.241642 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:56.241644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:56.241729 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:56.242226 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:56.242229 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:56.242324 (+ 95us) client_negotiation.cc:769] Sending connection context
0810 03:53:56.242406 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 03:53:56.242537 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260810 03:53:56.242867 25079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:56.230639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43634 (local address 127.27.153.66:34091)
0810 03:53:56.231105 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:56.231116 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:56.231140 (+ 24us) server_negotiation.cc:407] Connection header received
0810 03:53:56.231215 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:56.231221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:56.231432 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:56.233449 (+ 2017us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:56.233914 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.235144 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.236028 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.236654 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.236769 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:56.239042 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:56.239055 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:56.239074 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:56.239104 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:56.240947 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:56.241424 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:56.241430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:56.241432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:56.241533 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:56.241806 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:56.241809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:56.241811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:56.242127 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:56.242257 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:56.242503 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 03:53:56.242673 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":147,"threads_started":1}
I20260810 03:53:56.244035 24945 tablet_service.cc:1514] Processing DeleteTablet for tablet 12eaf68c75a94e2f961e27f37a03f059 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:53:35 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:43634
I20260810 03:53:56.244983 25081 tablet_replica.cc:330] stopping tablet replica
I20260810 03:53:56.245287 25081 raft_consensus.cc:2229] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:53:56.245536 25081 raft_consensus.cc:2258] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:53:56.246547 25081 ts_tablet_manager.cc:1905] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:53:56.250651 25081 ts_tablet_manager.cc:1918] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:53:56.250768 25081 log.cc:1196] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/12eaf68c75a94e2f961e27f37a03f059
I20260810 03:53:56.251036 25081 ts_tablet_manager.cc:1939] T 12eaf68c75a94e2f961e27f37a03f059 P caf97fd4c64e452aaa50b640477abb4a: Deleting consensus metadata
I20260810 03:53:56.252832 24455 catalog_manager.cc:4841] TS caf97fd4c64e452aaa50b640477abb4a (127.27.153.66:34091): tablet 12eaf68c75a94e2f961e27f37a03f059 (table test-table [id=0359e53ee72a4f7aa90091466178126e]) successfully deleted
W20260810 03:53:56.340147 25077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:53:56.340464 25077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:53:56.340517 25077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:53:56.344795 25077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:53:56.344866 25077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:53:56.344929 25077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:53:56.350481 25077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:43087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:33387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=43737
--tserver_master_addrs=127.27.153.126:42371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:53:56.351641 25077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:53:56.352674 25077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:53:56.362754 25094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:53:56.362766 25091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:56.362984 25077 server_base.cc:1034] running on GCE node
W20260810 03:53:56.362763 25088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:53:56.363595 25077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:53:56.364297 25077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:53:56.365471 25077 hybrid_clock.cc:648] HybridClock initialized: now 1786334036365452 us; error 30 us; skew 500 ppm
Aug 10 03:53:56 dist-test-slave-pww0 krb5kdc[23426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334036, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:53:56.368348 25077 init.cc:377] Logged in from keytab as kudu/127.27.153.67@KRBTEST.COM (short username kudu)
I20260810 03:53:56.369326 25077 webserver.cc:460] Webserver started at http://127.27.153.67:43737/ using document root <none> and password file <none>
I20260810 03:53:56.369901 25077 fs_manager.cc:359] Metadata directory not provided
I20260810 03:53:56.369994 25077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:53:56.374394 25077 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:53:56.376710 25105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:56.378051 25077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:53:56.378175 25077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "61990e39cfa44e9ca31271e185bc4787"
format_stamp: "Formatted at 2026-08-10 03:53:35 on dist-test-slave-pww0"
server_key: "e978b61dd2652c3473c9dd890c7b810a"
server_key_iv: "9ce4d17f0e519909a5de42d284425d72"
server_key_version: "encryptionkey@0"
I20260810 03:53:56.378747 25077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:53:56.409353 25077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:53:56.412731 25077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:53:56.412948 25077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:53:56.413580 25077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:53:56.414463 25077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:53:56.414515 25077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:56.414573 25077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:53:56.414608 25077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:53:56.424024 25077 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:33387
I20260810 03:53:56.424055 25301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:33387 every 8 connection(s)
I20260810 03:53:56.425081 25077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
Aug 10 03:53:56 dist-test-slave-pww0 krb5kdc[23426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334036, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:53:56.435444 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 25077
I20260810 03:53:56.439762 24818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:56.426969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:38635 (local address 127.27.153.126:42371)
0810 03:53:56.427145 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:56.427150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:56.428103 (+ 953us) server_negotiation.cc:407] Connection header received
0810 03:53:56.429403 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:56.429406 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:56.429463 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:56.429545 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:56.430825 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.431383 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.432688 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.432920 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.433032 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:56.435567 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:56.435572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:56.435575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:56.435612 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:56.437475 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:56.438072 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:56.438076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:56.438077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:56.438173 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:56.438555 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:56.438559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:56.438561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:56.438758 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:56.438872 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:56.439395 (+ 523us) server_negotiation.cc:299] Negotiation successful
0810 03:53:56.439573 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 03:53:56.440745 25307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:56.427408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.67:38635)
0810 03:53:56.427942 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:56.427987 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:56.429168 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:56.429742 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:56.429749 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:56.430125 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:53:56.430654 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:56.430669 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.431560 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.431563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:56.432541 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:56.432557 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.433047 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.433056 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:56.433201 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:56.433798 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:53:56.433814 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:53:56.435310 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:53:56.437635 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:56.437641 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:56.437654 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:56.437958 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:56.438264 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:53:56.438267 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:53:56.438269 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:53:56.438461 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:53:56.438875 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:53:56.438881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:53:56.439149 (+ 268us) client_negotiation.cc:769] Sending connection context
0810 03:53:56.439313 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 03:53:56.439649 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":117,"threads_started":1}
I20260810 03:53:56.442209 25302 heartbeater.cc:347] Connected to a master server at 127.27.153.126:42371
I20260810 03:53:56.442487 25302 heartbeater.cc:464] Registering TS with master...
I20260810 03:53:56.443065 25302 heartbeater.cc:511] Master 127.27.153.126:42371 requested a full tablet report, sending...
I20260810 03:53:56.444222 24481 ts_manager.cc:194] Registered new tserver with Master: 61990e39cfa44e9ca31271e185bc4787 (127.27.153.67:33387)
I20260810 03:53:56.444788 24481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:38635
I20260810 03:53:56.445307 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:56.436312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37322 (local address 127.27.153.126:42371)
0810 03:53:56.436584 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:56.436588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:56.436749 (+ 161us) server_negotiation.cc:407] Connection header received
0810 03:53:56.436895 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:56.436897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:56.436937 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:56.437013 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:56.437973 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.438710 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.439796 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.440134 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.440237 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:56.441864 (+ 1627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:56.441868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:56.441871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:56.441897 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:56.443397 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:56.443879 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:56.443883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:56.443885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:56.443968 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:56.444259 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:56.444262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:56.444264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:56.444503 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:56.444760 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:56.445005 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 03:53:56.445132 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":190,"thread_start_us":399,"threads_started":1}
I20260810 03:53:56.447279 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:53:56.451324 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:56.447953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37330 (local address 127.27.153.126:42371)
0810 03:53:56.448069 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:56.448073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:56.448264 (+ 191us) server_negotiation.cc:407] Connection header received
0810 03:53:56.448406 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:56.448408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:56.448508 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:56.448624 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:53:56.448850 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.449343 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.450201 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.450435 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.450522 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:56.450816 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:53:56.450949 (+ 133us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 03:53:56.451196 (+ 247us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37330: token expired
Metrics: {"server-negotiator.queue_time_us":32}
W20260810 03:53:56.451387 25308 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37330: token expired
W20260810 03:53:56.451839 24345 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.153.126:42371)
I20260810 03:53:56.459154 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:56.452211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37340 (local address 127.27.153.126:42371)
0810 03:53:56.452343 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:56.452347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:56.452362 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:53:56.452438 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:56.452440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:56.452482 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:56.452549 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:53:56.453138 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.453620 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.454376 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.454609 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.454699 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:56.456112 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:53:56.456115 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:53:56.456117 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:53:56.456141 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:53:56.457515 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:56.458002 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:56.458005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:56.458006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:56.458132 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:53:56.458386 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:53:56.458389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:53:56.458390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:53:56.458594 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:53:56.458698 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:56.458876 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 03:53:56.458979 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 03:53:56.462028 24481 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:53:56.465581 24481 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:53:56.473836 24698 tablet_service.cc:1467] Processing CreateTablet for tablet 46fd6ebf839e495fae99efacbdf2d568 (DEFAULT_TABLE table=test-table [id=0a4c62c466f64e29926438a0aba2d7ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:53:56.473878 24945 tablet_service.cc:1467] Processing CreateTablet for tablet 46fd6ebf839e495fae99efacbdf2d568 (DEFAULT_TABLE table=test-table [id=0a4c62c466f64e29926438a0aba2d7ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:53:56.474565 24698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46fd6ebf839e495fae99efacbdf2d568. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:53:56.474557 24945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46fd6ebf839e495fae99efacbdf2d568. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:53:56.477706 25319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:56.473100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54760 (local address 127.27.153.67:33387)
0810 03:53:56.473501 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:56.473514 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:56.473540 (+ 26us) server_negotiation.cc:407] Connection header received
0810 03:53:56.473619 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:56.473623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:56.473806 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:56.473955 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:53:56.474194 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.475027 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.476492 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.477079 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.477193 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:56.477371 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:56.477471 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 03:53:56.477572 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":158,"threads_started":1}
I20260810 03:53:56.477694 24826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:56.472823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:33387 (local address 127.0.0.1:54760)
0810 03:53:56.473092 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:56.473107 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:56.473175 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:56.473952 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:56.473955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:56.473974 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:53:56.474081 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:56.474089 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.475283 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.475287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:56.476216 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:56.476228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.477207 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.477211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:56.477304 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:56.477356 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 03:53:56.477438 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 03:53:56.477542 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190}
I20260810 03:53:56.477707 24870 tablet_bootstrap.cc:492] T 46fd6ebf839e495fae99efacbdf2d568 P caf97fd4c64e452aaa50b640477abb4a: Bootstrap starting.
I20260810 03:53:56.478199 24619 tablet_bootstrap.cc:492] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Bootstrap starting.
I20260810 03:53:56.478833 24870 tablet_bootstrap.cc:654] T 46fd6ebf839e495fae99efacbdf2d568 P caf97fd4c64e452aaa50b640477abb4a: Neither blocks nor log segments found. Creating new log.
I20260810 03:53:56.479064 24619 tablet_bootstrap.cc:654] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Neither blocks nor log segments found. Creating new log.
I20260810 03:53:56.480789 24870 tablet_bootstrap.cc:492] T 46fd6ebf839e495fae99efacbdf2d568 P caf97fd4c64e452aaa50b640477abb4a: No bootstrap required, opened a new log
I20260810 03:53:56.480899 24870 ts_tablet_manager.cc:1397] T 46fd6ebf839e495fae99efacbdf2d568 P caf97fd4c64e452aaa50b640477abb4a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20260810 03:53:56.481058 24619 tablet_bootstrap.cc:492] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547: No bootstrap required, opened a new log
I20260810 03:53:56.481156 24619 ts_tablet_manager.cc:1397] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260810 03:53:56.481426 24870 raft_consensus.cc:348] T 46fd6ebf839e495fae99efacbdf2d568 P caf97fd4c64e452aaa50b640477abb4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } }
I20260810 03:53:56.481568 24870 raft_consensus.cc:374] T 46fd6ebf839e495fae99efacbdf2d568 P caf97fd4c64e452aaa50b640477abb4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:53:56.481622 24870 raft_consensus.cc:729] T 46fd6ebf839e495fae99efacbdf2d568 P caf97fd4c64e452aaa50b640477abb4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caf97fd4c64e452aaa50b640477abb4a, State: Initialized, Role: FOLLOWER
I20260810 03:53:56.481575 24619 raft_consensus.cc:348] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } }
I20260810 03:53:56.481691 24619 raft_consensus.cc:374] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:53:56.481734 24619 raft_consensus.cc:729] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcfc8fa8c7ac40dbb3ee8f78d7270547, State: Initialized, Role: FOLLOWER
I20260810 03:53:56.481802 24870 consensus_queue.cc:260] T 46fd6ebf839e495fae99efacbdf2d568 P caf97fd4c64e452aaa50b640477abb4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } }
I20260810 03:53:56.481873 24619 consensus_queue.cc:260] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } }
I20260810 03:53:56.482148 24619 ts_tablet_manager.cc:1428] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 03:53:56.481832 25179 tablet_service.cc:1467] Processing CreateTablet for tablet 46fd6ebf839e495fae99efacbdf2d568 (DEFAULT_TABLE table=test-table [id=0a4c62c466f64e29926438a0aba2d7ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:53:56.483179 25179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46fd6ebf839e495fae99efacbdf2d568. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:53:56.483300 24870 ts_tablet_manager.cc:1428] T 46fd6ebf839e495fae99efacbdf2d568 P caf97fd4c64e452aaa50b640477abb4a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260810 03:53:56.486936 24817 tablet.cc:2369] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:53:56.491057 25326 tablet_bootstrap.cc:492] T 46fd6ebf839e495fae99efacbdf2d568 P 61990e39cfa44e9ca31271e185bc4787: Bootstrap starting.
I20260810 03:53:56.493438 25326 tablet_bootstrap.cc:654] T 46fd6ebf839e495fae99efacbdf2d568 P 61990e39cfa44e9ca31271e185bc4787: Neither blocks nor log segments found. Creating new log.
I20260810 03:53:56.494241 25326 log.cc:824] T 46fd6ebf839e495fae99efacbdf2d568 P 61990e39cfa44e9ca31271e185bc4787: Log is configured to *not* fsync() on all Append() calls
I20260810 03:53:56.496484 25326 tablet_bootstrap.cc:492] T 46fd6ebf839e495fae99efacbdf2d568 P 61990e39cfa44e9ca31271e185bc4787: No bootstrap required, opened a new log
I20260810 03:53:56.496713 25326 ts_tablet_manager.cc:1397] T 46fd6ebf839e495fae99efacbdf2d568 P 61990e39cfa44e9ca31271e185bc4787: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:53:56.500109 25326 raft_consensus.cc:348] T 46fd6ebf839e495fae99efacbdf2d568 P 61990e39cfa44e9ca31271e185bc4787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } }
I20260810 03:53:56.500385 25326 raft_consensus.cc:374] T 46fd6ebf839e495fae99efacbdf2d568 P 61990e39cfa44e9ca31271e185bc4787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:53:56.500456 25326 raft_consensus.cc:729] T 46fd6ebf839e495fae99efacbdf2d568 P 61990e39cfa44e9ca31271e185bc4787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61990e39cfa44e9ca31271e185bc4787, State: Initialized, Role: FOLLOWER
I20260810 03:53:56.500998 25326 consensus_queue.cc:260] T 46fd6ebf839e495fae99efacbdf2d568 P 61990e39cfa44e9ca31271e185bc4787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } }
I20260810 03:53:56.501950 25302 heartbeater.cc:503] Master 127.27.153.126:42371 was elected leader, sending a full tablet report...
I20260810 03:53:56.502054 25326 ts_tablet_manager.cc:1428] T 46fd6ebf839e495fae99efacbdf2d568 P 61990e39cfa44e9ca31271e185bc4787: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 03:53:56.604610 24825 raft_consensus.cc:482] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:53:56.604861 24825 raft_consensus.cc:504] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } }
I20260810 03:53:56.605949 24825 leader_election.cc:288] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61990e39cfa44e9ca31271e185bc4787 (127.27.153.67:33387), caf97fd4c64e452aaa50b640477abb4a (127.27.153.66:34091)
I20260810 03:53:56.610210 25079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:56.606332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:44947 (local address 127.27.153.66:34091)
0810 03:53:56.606474 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:56.606478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:56.606837 (+ 359us) server_negotiation.cc:407] Connection header received
0810 03:53:56.606957 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:56.606961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:56.607019 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:56.607105 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:53:56.607419 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.608016 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.609241 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.609703 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.609815 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:56.609975 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:56.610008 (+ 33us) server_negotiation.cc:299] Negotiation successful
0810 03:53:56.610076 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 03:53:56.610224 25319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:56.606370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:40499 (local address 127.27.153.67:33387)
0810 03:53:56.606537 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:56.606541 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:56.606718 (+ 177us) server_negotiation.cc:407] Connection header received
0810 03:53:56.606901 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:56.606903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:56.606976 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:56.607044 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:53:56.607350 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.607971 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.609237 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.609652 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.609767 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:56.609824 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:56.610024 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 03:53:56.610099 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 03:53:56.610230 25337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:56.606232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:34091 (local address 127.27.153.65:44947)
0810 03:53:56.606733 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:56.606747 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:56.606863 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:56.607130 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:56.607134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:56.607150 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:53:56.607246 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:56.607252 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.608150 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.608154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:56.609089 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:56.609100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.609819 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.609822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:56.609903 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:56.609925 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 03:53:56.609983 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 03:53:56.610065 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":59,"thread_start_us":81,"threads_started":1}
I20260810 03:53:56.610234 25336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:56.606233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:33387 (local address 127.27.153.65:40499)
0810 03:53:56.606598 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:56.606611 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:53:56.606755 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:53:56.607066 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:53:56.607069 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:53:56.607087 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:53:56.607222 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:56.607228 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.608078 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.608081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:56.609088 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:53:56.609099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.609767 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.609771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:53:56.609866 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:56.609920 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 03:53:56.609989 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 03:53:56.610072 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":55,"thread_start_us":98,"threads_started":1}
I20260810 03:53:56.610968 25218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46fd6ebf839e495fae99efacbdf2d568" candidate_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61990e39cfa44e9ca31271e185bc4787" is_pre_election: true
I20260810 03:53:56.610968 24985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46fd6ebf839e495fae99efacbdf2d568" candidate_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caf97fd4c64e452aaa50b640477abb4a" is_pre_election: true
I20260810 03:53:56.611310 24985 raft_consensus.cc:2454] T 46fd6ebf839e495fae99efacbdf2d568 P caf97fd4c64e452aaa50b640477abb4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcfc8fa8c7ac40dbb3ee8f78d7270547 in term 0.
I20260810 03:53:56.611310 25218 raft_consensus.cc:2454] T 46fd6ebf839e495fae99efacbdf2d568 P 61990e39cfa44e9ca31271e185bc4787 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fcfc8fa8c7ac40dbb3ee8f78d7270547 in term 0.
I20260810 03:53:56.611886 24606 leader_election.cc:302] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 61990e39cfa44e9ca31271e185bc4787, caf97fd4c64e452aaa50b640477abb4a, fcfc8fa8c7ac40dbb3ee8f78d7270547; no voters:
I20260810 03:53:56.612210 24825 raft_consensus.cc:2781] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:53:56.612300 24825 raft_consensus.cc:482] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:53:56.612350 24825 raft_consensus.cc:3037] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:53:56.613443 24825 raft_consensus.cc:504] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } }
I20260810 03:53:56.613817 24825 leader_election.cc:288] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 1 election: Requested vote from peers 61990e39cfa44e9ca31271e185bc4787 (127.27.153.67:33387), caf97fd4c64e452aaa50b640477abb4a (127.27.153.66:34091)
I20260810 03:53:56.614256 24985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46fd6ebf839e495fae99efacbdf2d568" candidate_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caf97fd4c64e452aaa50b640477abb4a"
I20260810 03:53:56.614310 25218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46fd6ebf839e495fae99efacbdf2d568" candidate_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61990e39cfa44e9ca31271e185bc4787"
I20260810 03:53:56.614388 24985 raft_consensus.cc:3037] T 46fd6ebf839e495fae99efacbdf2d568 P caf97fd4c64e452aaa50b640477abb4a [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:53:56.614432 25218 raft_consensus.cc:3037] T 46fd6ebf839e495fae99efacbdf2d568 P 61990e39cfa44e9ca31271e185bc4787 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:53:56.615536 25218 raft_consensus.cc:2454] T 46fd6ebf839e495fae99efacbdf2d568 P 61990e39cfa44e9ca31271e185bc4787 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcfc8fa8c7ac40dbb3ee8f78d7270547 in term 1.
I20260810 03:53:56.615540 24985 raft_consensus.cc:2454] T 46fd6ebf839e495fae99efacbdf2d568 P caf97fd4c64e452aaa50b640477abb4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fcfc8fa8c7ac40dbb3ee8f78d7270547 in term 1.
I20260810 03:53:56.615866 24606 leader_election.cc:302] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: caf97fd4c64e452aaa50b640477abb4a, fcfc8fa8c7ac40dbb3ee8f78d7270547; no voters:
I20260810 03:53:56.616046 24825 raft_consensus.cc:2781] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:53:56.616257 24825 raft_consensus.cc:686] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 LEADER]: Becoming Leader. State: Replica: fcfc8fa8c7ac40dbb3ee8f78d7270547, State: Running, Role: LEADER
I20260810 03:53:56.616542 24825 consensus_queue.cc:237] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } }
I20260810 03:53:56.619758 24479 catalog_manager.cc:5495] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 reported cstate change: term changed from 0 to 1, leader changed from <none> to fcfc8fa8c7ac40dbb3ee8f78d7270547 (127.27.153.65). New cstate: current_term: 1 leader_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } health_report { overall_health: UNKNOWN } } }
I20260810 03:53:56.637691 25343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:56.633416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56654 (local address 127.27.153.65:38975)
0810 03:53:56.633770 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0810 03:53:56.633778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:53:56.633792 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:53:56.633876 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:53:56.633878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:53:56.633934 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:53:56.634054 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:53:56.634335 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.635140 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.636114 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:53:56.636377 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:53:56.636470 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:53:56.636793 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:53:56.637259 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:53:56.637350 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 03:53:56.637481 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 03:53:56.637547 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":140,"threads_started":1}
I20260810 03:53:56.642346 25218 raft_consensus.cc:1261] T 46fd6ebf839e495fae99efacbdf2d568 P 61990e39cfa44e9ca31271e185bc4787 [term 1 FOLLOWER]: Refusing update from remote peer fcfc8fa8c7ac40dbb3ee8f78d7270547: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:53:56.642341 24985 raft_consensus.cc:1261] T 46fd6ebf839e495fae99efacbdf2d568 P caf97fd4c64e452aaa50b640477abb4a [term 1 FOLLOWER]: Refusing update from remote peer fcfc8fa8c7ac40dbb3ee8f78d7270547: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:53:56.643115 25338 consensus_queue.cc:1035] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:53:56.643313 24825 consensus_queue.cc:1035] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:53:56.648804 25344 mvcc.cc:205] Tried to move back new op lower bound from 7316824214079098880 to 7316824213984243712. Current Snapshot: MvccSnapshot[applied={T|T < 7316824214079098880}]
I20260810 03:53:56.650074 25346 mvcc.cc:205] Tried to move back new op lower bound from 7316824214079098880 to 7316824213984243712. Current Snapshot: MvccSnapshot[applied={T|T < 7316824214079098880}]
I20260810 03:53:56.651899 25348 mvcc.cc:205] Tried to move back new op lower bound from 7316824214079098880 to 7316824213984243712. Current Snapshot: MvccSnapshot[applied={T|T < 7316824214079098880}]
I20260810 03:53:56.659013 24481 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:53:56.659237 24481 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:53:56.662065 24481 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be: Sending DeleteTablet for 3 replicas of tablet 46fd6ebf839e495fae99efacbdf2d568
I20260810 03:53:56.662976 24698 tablet_service.cc:1514] Processing DeleteTablet for tablet 46fd6ebf839e495fae99efacbdf2d568 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:53:56 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:56638
I20260810 03:53:56.662904 25179 tablet_service.cc:1514] Processing DeleteTablet for tablet 46fd6ebf839e495fae99efacbdf2d568 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:53:56 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:54760
I20260810 03:53:56.663013 24945 tablet_service.cc:1514] Processing DeleteTablet for tablet 46fd6ebf839e495fae99efacbdf2d568 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:53:56 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:43634
I20260810 03:53:56.663120 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 24546
I20260810 03:53:56.666337 25081 tablet_replica.cc:330] T 46fd6ebf839e495fae99efacbdf2d568 P caf97fd4c64e452aaa50b640477abb4a: stopping tablet replica
I20260810 03:53:56.666507 25081 raft_consensus.cc:2229] T 46fd6ebf839e495fae99efacbdf2d568 P caf97fd4c64e452aaa50b640477abb4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:53:56.666667 25081 raft_consensus.cc:2258] T 46fd6ebf839e495fae99efacbdf2d568 P caf97fd4c64e452aaa50b640477abb4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:53:56.674226 25081 ts_tablet_manager.cc:1905] T 46fd6ebf839e495fae99efacbdf2d568 P caf97fd4c64e452aaa50b640477abb4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:53:56.676370 25366 tablet_replica.cc:330] T 46fd6ebf839e495fae99efacbdf2d568 P 61990e39cfa44e9ca31271e185bc4787: stopping tablet replica
I20260810 03:53:56.677150 25081 ts_tablet_manager.cc:1918] T 46fd6ebf839e495fae99efacbdf2d568 P caf97fd4c64e452aaa50b640477abb4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:53:56.677174 25366 raft_consensus.cc:2229] T 46fd6ebf839e495fae99efacbdf2d568 P 61990e39cfa44e9ca31271e185bc4787 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:53:56.677248 25081 log.cc:1196] T 46fd6ebf839e495fae99efacbdf2d568 P caf97fd4c64e452aaa50b640477abb4a: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/46fd6ebf839e495fae99efacbdf2d568
I20260810 03:53:56.677488 25081 ts_tablet_manager.cc:1939] T 46fd6ebf839e495fae99efacbdf2d568 P caf97fd4c64e452aaa50b640477abb4a: Deleting consensus metadata
I20260810 03:53:56.677559 25366 raft_consensus.cc:2258] T 46fd6ebf839e495fae99efacbdf2d568 P 61990e39cfa44e9ca31271e185bc4787 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:53:56.678357 24455 catalog_manager.cc:4841] TS caf97fd4c64e452aaa50b640477abb4a (127.27.153.66:34091): tablet 46fd6ebf839e495fae99efacbdf2d568 (table test-table [id=0a4c62c466f64e29926438a0aba2d7ab]) successfully deleted
I20260810 03:53:56.679446 25366 ts_tablet_manager.cc:1905] T 46fd6ebf839e495fae99efacbdf2d568 P 61990e39cfa44e9ca31271e185bc4787: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:53:56.682304 25366 ts_tablet_manager.cc:1918] T 46fd6ebf839e495fae99efacbdf2d568 P 61990e39cfa44e9ca31271e185bc4787: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:53:56.682421 25366 log.cc:1196] T 46fd6ebf839e495fae99efacbdf2d568 P 61990e39cfa44e9ca31271e185bc4787: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/46fd6ebf839e495fae99efacbdf2d568
I20260810 03:53:56.682773 25366 ts_tablet_manager.cc:1939] T 46fd6ebf839e495fae99efacbdf2d568 P 61990e39cfa44e9ca31271e185bc4787: Deleting consensus metadata
I20260810 03:53:56.683310 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 24822
W20260810 03:53:56.683450 24455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.153.65:38975 (error 108)
I20260810 03:53:56.692391 24459 catalog_manager.cc:4841] TS 61990e39cfa44e9ca31271e185bc4787 (127.27.153.67:33387): tablet 46fd6ebf839e495fae99efacbdf2d568 (table test-table [id=0a4c62c466f64e29926438a0aba2d7ab]) successfully deleted
I20260810 03:53:56.694397 24826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:53:56.694088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:38975 (local address 127.0.0.1:56658)
0810 03:53:56.694223 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 03:53:56.694310 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:38975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 03:53:56.694585 24455 catalog_manager.cc:4584] TS fcfc8fa8c7ac40dbb3ee8f78d7270547 (127.27.153.65:38975): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 46fd6ebf839e495fae99efacbdf2d568: Network error: Client connection negotiation failed: client connection to 127.27.153.65:38975: connect: Connection refused (error 111)
I20260810 03:53:56.698704 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 25077
I20260810 03:53:56.722744 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 24422
I20260810 03:54:16.742479 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:42371
--webserver_interface=127.27.153.126
--webserver_port=41403
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:43087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:42371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:54:16.862443 25373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:16.862752 25373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:16.862808 25373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:16.866978 25373 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 03:54:16.867053 25373 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 03:54:16.867069 25373 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 03:54:16.867082 25373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:54:16.867100 25373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:16.867117 25373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:16.867136 25373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:54:16.867167 25373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:54:16.872829 25373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:43087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:42371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:42371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=41403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:16.874018 25373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:16.875095 25373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:16.884455 25382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:16.884461 25383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:16.884461 25373 server_base.cc:1034] running on GCE node
W20260810 03:54:16.884517 25388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:16.885375 25373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:16.886554 25373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:16.887724 25373 hybrid_clock.cc:648] HybridClock initialized: now 1786334056887704 us; error 36 us; skew 500 ppm
Aug 10 03:54:16 dist-test-slave-pww0 krb5kdc[23426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334056, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:16.890498 25373 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:54:16.891506 25373 webserver.cc:460] Webserver started at http://127.27.153.126:41403/ using document root <none> and password file <none>
I20260810 03:54:16.892071 25373 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:16.892139 25373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:16.896337 25373 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:54:16.899735 25398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:16.901028 25373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:54:16.901153 25373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "356ae6557eff400d8b1de99f512af1be"
format_stamp: "Formatted at 2026-08-10 03:53:34 on dist-test-slave-pww0"
server_key: "ff91e49bdb94950e2014f72683062e60"
server_key_iv: "cc8f5d442a8dbc83e604b1f755b7b470"
server_key_version: "encryptionkey@0"
I20260810 03:54:16.901741 25373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:16.918519 25373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:16.921654 25373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:16.921861 25373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:16.929065 25373 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:42371
I20260810 03:54:16.929414 25509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:42371 every 8 connection(s)
I20260810 03:54:16.930181 25373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:54:16.931216 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 25373
I20260810 03:54:16.931758 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:38975
--local_ip_for_outbound_sockets=127.27.153.65
--tserver_master_addrs=127.27.153.126:42371
--webserver_port=38379
--webserver_interface=127.27.153.65
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:43087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:54:16.935535 25510 sys_catalog.cc:263] Verifying existing consensus state
I20260810 03:54:16.937356 25510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be: Bootstrap starting.
I20260810 03:54:16.955399 25510 log.cc:824] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:16.967859 25510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:54:16.968273 25510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be: Bootstrap complete.
I20260810 03:54:16.972414 25510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356ae6557eff400d8b1de99f512af1be" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42371 } }
I20260810 03:54:16.972836 25510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 356ae6557eff400d8b1de99f512af1be, State: Initialized, Role: FOLLOWER
I20260810 03:54:16.973361 25510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356ae6557eff400d8b1de99f512af1be" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42371 } }
I20260810 03:54:16.973487 25510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:54:16.973552 25510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:54:16.973654 25510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 2 FOLLOWER]: Advancing to term 3
I20260810 03:54:16.975720 25510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356ae6557eff400d8b1de99f512af1be" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42371 } }
I20260810 03:54:16.976055 25510 leader_election.cc:302] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 356ae6557eff400d8b1de99f512af1be; no voters:
I20260810 03:54:16.976375 25510 leader_election.cc:288] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 03:54:16.976482 25517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 3 FOLLOWER]: Leader election won for term 3
I20260810 03:54:16.976684 25517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [term 3 LEADER]: Becoming Leader. State: Replica: 356ae6557eff400d8b1de99f512af1be, State: Running, Role: LEADER
I20260810 03:54:16.976995 25517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356ae6557eff400d8b1de99f512af1be" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42371 } }
I20260810 03:54:16.977727 25510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:54:16.978673 25522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 356ae6557eff400d8b1de99f512af1be. Latest consensus state: current_term: 3 leader_uuid: "356ae6557eff400d8b1de99f512af1be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356ae6557eff400d8b1de99f512af1be" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42371 } } }
I20260810 03:54:16.978685 25518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "356ae6557eff400d8b1de99f512af1be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "356ae6557eff400d8b1de99f512af1be" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42371 } } }
I20260810 03:54:16.978824 25522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:16.978833 25518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:16.979219 25533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:54:16.984593 25533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:54:16.985330 25533 catalog_manager.cc:1259] Loaded cluster ID: b77c67349c6342e4932dc89ba6409c7d
I20260810 03:54:16.985399 25533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:54:16.988065 25533 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:54:16.988878 25533 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be: Loaded TSK: 0
I20260810 03:54:16.988974 25533 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 356ae6557eff400d8b1de99f512af1be: Loaded TSK: 1
F20260810 03:54:16.989784 25533 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f696704e9ad google::LogMessage::Fail() at ??:0
@ 0x7f696705188a google::LogMessage::SendToLog() at ??:0
@ 0x7f696704e4de google::LogMessage::Flush() at ??:0
@ 0x7f696704fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6969d1218d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f6969d0af0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f69719b77bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f697199199c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f6971992178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f69719d3df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f69719f282b std::function<>::operator()() at ??:0
@ 0x7f6971991b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f6971992f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f697198f579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f69719d3640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f69692a03d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f69692a0c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f69692a23f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6969291512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f696a8d56db start_thread at ??:0
@ 0x7f6965b4c61f clone at ??:0
W20260810 03:54:17.076331 25514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:17.076685 25514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:17.076751 25514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:17.082996 25514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:17.083073 25514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:17.083165 25514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:54:17.091516 25514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:43087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:38975
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=38379
--tserver_master_addrs=127.27.153.126:42371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:17.093129 25514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:17.094249 25514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:17.104571 25595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:17.104733 25603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:17.104571 25589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:17.105347 25514 server_base.cc:1034] running on GCE node
I20260810 03:54:17.105857 25514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:17.106576 25514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:17.107753 25514 hybrid_clock.cc:648] HybridClock initialized: now 1786334057107737 us; error 34 us; skew 500 ppm
Aug 10 03:54:17 dist-test-slave-pww0 krb5kdc[23426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334057, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:17.110587 25514 init.cc:377] Logged in from keytab as kudu/127.27.153.65@KRBTEST.COM (short username kudu)
I20260810 03:54:17.111668 25514 webserver.cc:460] Webserver started at http://127.27.153.65:38379/ using document root <none> and password file <none>
I20260810 03:54:17.112246 25514 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:17.112313 25514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:17.116427 25514 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:54:17.118721 25614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:17.120091 25514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:54:17.120205 25514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547"
format_stamp: "Formatted at 2026-08-10 03:53:34 on dist-test-slave-pww0"
server_key: "ffc2269d8c50241c17c7978e2a131963"
server_key_iv: "91b3579162793ec72b95edd75cadfc68"
server_key_version: "encryptionkey@0"
I20260810 03:54:17.120787 25514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:17.145334 25514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:17.148902 25514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:17.149091 25514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:17.149776 25514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:54:17.150997 25624 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 03:54:17.157050 25514 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 03:54:17.157137 25514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260810 03:54:17.157212 25514 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 03:54:17.159119 25514 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 03:54:17.159191 25514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 03:54:17.159557 25624 tablet_bootstrap.cc:492] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Bootstrap starting.
I20260810 03:54:17.169493 25514 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:38975
I20260810 03:54:17.169801 25834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:38975 every 8 connection(s)
I20260810 03:54:17.170635 25514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:54:17.176472 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 25514
I20260810 03:54:17.177012 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:34091
--local_ip_for_outbound_sockets=127.27.153.66
--tserver_master_addrs=127.27.153.126:42371
--webserver_port=36955
--webserver_interface=127.27.153.66
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:43087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:54:17.181242 25624 log.cc:824] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:17.188820 25624 tablet_bootstrap.cc:492] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 03:54:17.189227 25624 tablet_bootstrap.cc:492] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Bootstrap complete.
I20260810 03:54:17.190318 25624 ts_tablet_manager.cc:1397] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Time spent bootstrapping tablet: real 0.031s user 0.018s sys 0.010s
I20260810 03:54:17.192902 25624 raft_consensus.cc:348] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } }
I20260810 03:54:17.193425 25624 raft_consensus.cc:729] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcfc8fa8c7ac40dbb3ee8f78d7270547, State: Initialized, Role: FOLLOWER
I20260810 03:54:17.193954 25624 consensus_queue.cc:260] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } }
I20260810 03:54:17.194976 25624 ts_tablet_manager.cc:1428] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
W20260810 03:54:17.234782 25839 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:54:17.173214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.65:43397)
0810 03:54:17.173884 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:17.173952 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:17.175140 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:17.233957 (+ 58817us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":173,"threads_started":1}
W20260810 03:54:17.235428 25837 heartbeater.cc:644] Failed to heartbeat to 127.27.153.126:42371 (0 consecutive failures): Network error: Failed to ping master at 127.27.153.126:42371: Client connection negotiation failed: client connection to 127.27.153.126:42371: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 03:54:17.236348 25839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:17.236069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.65:45357)
0810 03:54:17.236205 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:17.236282 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 03:54:17.236977 25839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:17.236780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.65:48533)
0810 03:54:17.236881 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:17.236925 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 03:54:17.237138 25837 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:54:17.312201 25844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:17.312489 25844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:17.312532 25844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:17.316741 25844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:17.316815 25844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:17.316887 25844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:54:17.322355 25844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:43087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:34091
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=36955
--tserver_master_addrs=127.27.153.126:42371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:17.323580 25844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:17.324681 25844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:17.335099 25855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:17.335183 25844 server_base.cc:1034] running on GCE node
W20260810 03:54:17.335105 25856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:17.335093 25858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:17.335868 25844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:17.336580 25844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:17.337774 25844 hybrid_clock.cc:648] HybridClock initialized: now 1786334057337754 us; error 32 us; skew 500 ppm
Aug 10 03:54:17 dist-test-slave-pww0 krb5kdc[23426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334057, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:17.340687 25844 init.cc:377] Logged in from keytab as kudu/127.27.153.66@KRBTEST.COM (short username kudu)
I20260810 03:54:17.341715 25844 webserver.cc:460] Webserver started at http://127.27.153.66:36955/ using document root <none> and password file <none>
I20260810 03:54:17.342315 25844 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:17.342391 25844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:17.346661 25844 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:54:17.349046 25866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:17.350442 25844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:54:17.350598 25844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "caf97fd4c64e452aaa50b640477abb4a"
format_stamp: "Formatted at 2026-08-10 03:53:34 on dist-test-slave-pww0"
server_key: "60088d7a7f98ff7326cff790a6f3c4c4"
server_key_iv: "483b0cfd0eebae5766cd695cbfe98420"
server_key_version: "encryptionkey@0"
I20260810 03:54:17.351248 25844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:17.377542 25844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:17.381095 25844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:17.381318 25844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:17.381984 25844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:54:17.382870 25844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:54:17.382920 25844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:17.382987 25844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:54:17.383014 25844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:17.392077 25844 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:34091
I20260810 03:54:17.392112 26063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:34091 every 8 connection(s)
I20260810 03:54:17.393129 25844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:54:17.394644 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 25844
I20260810 03:54:17.395057 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:33387
--local_ip_for_outbound_sockets=127.27.153.67
--tserver_master_addrs=127.27.153.126:42371
--webserver_port=43737
--webserver_interface=127.27.153.67
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:43087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:54:17.398635 26066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:17.395497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.66:60273)
0810 03:54:17.397513 (+ 2016us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:17.397664 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1769,"thread_start_us":141,"threads_started":1}
W20260810 03:54:17.399091 26064 heartbeater.cc:644] Failed to heartbeat to 127.27.153.126:42371 (0 consecutive failures): Network error: Failed to ping master at 127.27.153.126:42371: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
I20260810 03:54:17.399822 26066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:17.399577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.66:45675)
0810 03:54:17.399705 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:17.399759 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 03:54:17.400413 26066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:17.400229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.66:50285)
0810 03:54:17.400320 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:17.400362 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 03:54:17.400604 26064 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 03:54:17.510694 26068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:17.510984 26068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:17.511049 26068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:17.515062 26068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:17.515132 26068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:17.515223 26068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:54:17.520659 26068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:43087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:33387
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=43737
--tserver_master_addrs=127.27.153.126:42371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:17.521772 26068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:17.522812 26068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:54:17.533133 26068 server_base.cc:1034] running on GCE node
W20260810 03:54:17.533088 26078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:17.533102 26076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:17.533087 26075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:17.533865 26068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:17.534585 26068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:17.535754 26068 hybrid_clock.cc:648] HybridClock initialized: now 1786334057535733 us; error 33 us; skew 500 ppm
Aug 10 03:54:17 dist-test-slave-pww0 krb5kdc[23426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334057, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:17.538564 26068 init.cc:377] Logged in from keytab as kudu/127.27.153.67@KRBTEST.COM (short username kudu)
I20260810 03:54:17.539582 26068 webserver.cc:460] Webserver started at http://127.27.153.67:43737/ using document root <none> and password file <none>
I20260810 03:54:17.540143 26068 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:17.540215 26068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:17.544307 26068 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260810 03:54:17.546536 26095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:17.548070 26068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:54:17.548211 26068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "61990e39cfa44e9ca31271e185bc4787"
format_stamp: "Formatted at 2026-08-10 03:53:35 on dist-test-slave-pww0"
server_key: "e978b61dd2652c3473c9dd890c7b810a"
server_key_iv: "9ce4d17f0e519909a5de42d284425d72"
server_key_version: "encryptionkey@0"
I20260810 03:54:17.548802 26068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:17.559985 26068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:17.563195 26068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:17.563401 26068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:17.564056 26068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:54:17.564956 26068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:54:17.565006 26068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:17.565063 26068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:54:17.565094 26068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:17.574280 26068 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:33387
I20260810 03:54:17.574360 26311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:33387 every 8 connection(s)
I20260810 03:54:17.575385 26068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:54:17.579553 26324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:17.577667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.67:57861)
0810 03:54:17.578233 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:17.578710 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":114,"threads_started":1}
W20260810 03:54:17.580112 26314 heartbeater.cc:644] Failed to heartbeat to 127.27.153.126:42371 (0 consecutive failures): Network error: Failed to ping master at 127.27.153.126:42371: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
I20260810 03:54:17.580802 26324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:17.580532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.67:50749)
0810 03:54:17.580677 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:17.580737 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 03:54:17.581396 26324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:17.581213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.67:40689)
0810 03:54:17.581300 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:17.581346 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 03:54:17.581563 26314 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 03:54:17.584431 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 26068
I20260810 03:54:18.238389 26326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:18.237845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.65:40079)
0810 03:54:18.238176 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:18.238248 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":110,"threads_started":1}
I20260810 03:54:18.401804 26327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:18.401271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.66:58737)
0810 03:54:18.401622 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:18.401691 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":101,"threads_started":1}
I20260810 03:54:18.582463 26328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:18.582062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.67:47537)
0810 03:54:18.582302 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:18.582354 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":70,"threads_started":1}
I20260810 03:54:19.043512 26332 raft_consensus.cc:482] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:54:19.043751 26332 raft_consensus.cc:504] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } }
I20260810 03:54:19.044888 26332 leader_election.cc:288] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61990e39cfa44e9ca31271e185bc4787 (127.27.153.67:33387), caf97fd4c64e452aaa50b640477abb4a (127.27.153.66:34091)
Aug 10 03:54:19 dist-test-slave-pww0 krb5kdc[23426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334057, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.67@KRBTEST.COM
Aug 10 03:54:19 dist-test-slave-pww0 krb5kdc[23426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334057, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.66@KRBTEST.COM
I20260810 03:54:19.062947 26339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:19.045200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:34091 (local address 127.27.153.65:41123)
0810 03:54:19.046161 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:19.046174 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:19.046288 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:19.047945 (+ 1657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:19.047952 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:19.048583 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:54:19.049405 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:19.049431 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:19.051573 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:19.051577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:19.053214 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:19.053232 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:19.054411 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:19.054416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:19.054785 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:19.055678 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:54:19.055714 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:54:19.057979 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:54:19.060295 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:54:19.060298 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:54:19.060299 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:54:19.060603 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:54:19.060960 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:54:19.060963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:54:19.060965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:54:19.061124 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:54:19.061860 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:54:19.061868 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:54:19.062136 (+ 268us) client_negotiation.cc:769] Sending connection context
0810 03:54:19.062314 (+ 178us) client_negotiation.cc:240] Negotiation successful
0810 03:54:19.062745 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":932,"spinlock_wait_cycles":21248,"thread_start_us":79,"threads_started":1}
I20260810 03:54:19.062949 26335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:19.045417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:41123 (local address 127.27.153.66:34091)
0810 03:54:19.046027 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:19.046036 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:19.046280 (+ 244us) server_negotiation.cc:407] Connection header received
0810 03:54:19.047064 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:19.047071 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:19.047441 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:19.047805 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:19.049670 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:19.051307 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:19.053418 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:19.054227 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:19.054434 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:19.058220 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:19.058225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:19.058238 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:19.058273 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:19.060154 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:19.060726 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:19.060731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:19.060735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:19.060862 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:19.061217 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:19.061220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:19.061222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:19.061726 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:19.061841 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:19.062466 (+ 625us) server_negotiation.cc:299] Negotiation successful
0810 03:54:19.062743 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":148,"threads_started":1}
I20260810 03:54:19.062948 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:19.045494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:44189 (local address 127.27.153.67:33387)
0810 03:54:19.045956 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:19.045990 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:19.046036 (+ 46us) server_negotiation.cc:407] Connection header received
0810 03:54:19.046943 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:19.046953 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:19.047441 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:19.047805 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:19.049670 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:19.051307 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:19.053451 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:19.054227 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:19.054434 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:19.057997 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:19.058006 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:19.058027 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:19.058069 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:19.060073 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:19.060734 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:19.060739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:19.060743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:19.060878 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:19.061233 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:19.061236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:19.061237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:19.061860 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:19.061999 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:19.062466 (+ 467us) server_negotiation.cc:299] Negotiation successful
0810 03:54:19.062764 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":101,"threads_started":1}
I20260810 03:54:19.062987 26333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:19.045200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:33387 (local address 127.27.153.65:44189)
0810 03:54:19.045652 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:19.045669 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:19.045784 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:19.047945 (+ 2161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:19.047952 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:19.048583 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:54:19.049405 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:19.049431 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:19.051573 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:19.051577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:19.053216 (+ 1639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:19.053232 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:19.054411 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:19.054416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:19.054782 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:19.055701 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:54:19.055718 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:54:19.057700 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:54:19.060219 (+ 2519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:54:19.060227 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:54:19.060243 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:54:19.060601 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:54:19.060978 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:54:19.060981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:54:19.060982 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:54:19.061134 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:54:19.061988 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:54:19.061990 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:54:19.062134 (+ 144us) client_negotiation.cc:769] Sending connection context
0810 03:54:19.062314 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 03:54:19.062823 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":377,"spinlock_wait_cycles":56320,"thread_start_us":108,"threads_started":1}
W20260810 03:54:19.065948 25615 leader_election.cc:341] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer caf97fd4c64e452aaa50b640477abb4a (127.27.153.66:34091): Not found: Tablet not found: 46fd6ebf839e495fae99efacbdf2d568
W20260810 03:54:19.066115 25620 leader_election.cc:341] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61990e39cfa44e9ca31271e185bc4787 (127.27.153.67:33387): Not found: Tablet not found: 46fd6ebf839e495fae99efacbdf2d568
I20260810 03:54:19.066188 25620 leader_election.cc:302] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fcfc8fa8c7ac40dbb3ee8f78d7270547; no voters: 61990e39cfa44e9ca31271e185bc4787, caf97fd4c64e452aaa50b640477abb4a
I20260810 03:54:19.066462 26332 raft_consensus.cc:2726] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:54:19.239665 26333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:19.239295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.65:34067)
0810 03:54:19.239503 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:19.239571 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 03:54:19.403146 26340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:19.402641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.66:48065)
0810 03:54:19.402948 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:19.403015 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":86,"threads_started":1}
I20260810 03:54:19.583681 26341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:19.583205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.67:49727)
0810 03:54:19.583516 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:19.583578 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":82,"threads_started":1}
I20260810 03:54:20.240968 26343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:20.240493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.65:43985)
0810 03:54:20.240793 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:20.240857 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":83,"threads_started":1}
I20260810 03:54:20.404580 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:20.404110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.66:56619)
0810 03:54:20.404424 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:20.404480 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":110,"threads_started":1}
I20260810 03:54:20.584911 26346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:20.584486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.67:49697)
0810 03:54:20.584745 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:20.584808 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":81,"threads_started":1}
I20260810 03:54:20.787878 26348 raft_consensus.cc:482] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:54:20.788028 26348 raft_consensus.cc:504] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } }
I20260810 03:54:20.788477 26348 leader_election.cc:288] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61990e39cfa44e9ca31271e185bc4787 (127.27.153.67:33387), caf97fd4c64e452aaa50b640477abb4a (127.27.153.66:34091)
W20260810 03:54:20.789412 25620 leader_election.cc:341] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61990e39cfa44e9ca31271e185bc4787 (127.27.153.67:33387): Not found: Tablet not found: 46fd6ebf839e495fae99efacbdf2d568
W20260810 03:54:20.789541 25615 leader_election.cc:341] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer caf97fd4c64e452aaa50b640477abb4a (127.27.153.66:34091): Not found: Tablet not found: 46fd6ebf839e495fae99efacbdf2d568
I20260810 03:54:20.789608 25615 leader_election.cc:302] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fcfc8fa8c7ac40dbb3ee8f78d7270547; no voters: 61990e39cfa44e9ca31271e185bc4787, caf97fd4c64e452aaa50b640477abb4a
I20260810 03:54:20.789753 26348 raft_consensus.cc:2726] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:54:21.242348 26350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:21.241853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.65:41265)
0810 03:54:21.242172 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:21.242245 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":84,"threads_started":1}
I20260810 03:54:21.405833 26353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:21.405360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.66:50311)
0810 03:54:21.405652 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:21.405729 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":81,"threads_started":1}
I20260810 03:54:21.586155 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:21.585719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.67:34825)
0810 03:54:21.585996 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:21.586056 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":76,"threads_started":1}
I20260810 03:54:22.243938 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:22.243400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.65:54765)
0810 03:54:22.243769 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:22.243837 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":83,"threads_started":1}
I20260810 03:54:22.407074 26357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:22.406642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.66:51097)
0810 03:54:22.406911 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:22.406973 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":88,"threads_started":1}
I20260810 03:54:22.587517 26358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:22.587014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.67:48411)
0810 03:54:22.587300 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:22.587376 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":89,"threads_started":1}
I20260810 03:54:23.245327 26360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:23.244824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.65:54427)
0810 03:54:23.245156 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:23.245227 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":104,"threads_started":1}
I20260810 03:54:23.408353 26362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:23.407890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.66:54065)
0810 03:54:23.408183 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:23.408249 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":79,"threads_started":1}
I20260810 03:54:23.588832 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:23.588365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.67:41119)
0810 03:54:23.588655 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:23.588718 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":99,"threads_started":1}
I20260810 03:54:23.591781 26367 raft_consensus.cc:482] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:54:23.591930 26367 raft_consensus.cc:504] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } }
I20260810 03:54:23.592402 26367 leader_election.cc:288] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61990e39cfa44e9ca31271e185bc4787 (127.27.153.67:33387), caf97fd4c64e452aaa50b640477abb4a (127.27.153.66:34091)
W20260810 03:54:23.593269 25620 leader_election.cc:341] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61990e39cfa44e9ca31271e185bc4787 (127.27.153.67:33387): Not found: Tablet not found: 46fd6ebf839e495fae99efacbdf2d568
W20260810 03:54:23.593456 25615 leader_election.cc:341] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer caf97fd4c64e452aaa50b640477abb4a (127.27.153.66:34091): Not found: Tablet not found: 46fd6ebf839e495fae99efacbdf2d568
I20260810 03:54:23.593560 25615 leader_election.cc:302] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fcfc8fa8c7ac40dbb3ee8f78d7270547; no voters: 61990e39cfa44e9ca31271e185bc4787, caf97fd4c64e452aaa50b640477abb4a
I20260810 03:54:23.593730 26367 raft_consensus.cc:2726] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:54:24.246749 26369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:24.246258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.65:34699)
0810 03:54:24.246559 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:24.246628 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":87,"threads_started":1}
I20260810 03:54:24.409762 26372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:24.409205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.66:56121)
0810 03:54:24.409568 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:24.409638 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":86,"threads_started":1}
I20260810 03:54:24.590082 26373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:24.589635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.67:58501)
0810 03:54:24.589928 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:24.589985 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":75,"threads_started":1}
I20260810 03:54:25.248219 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:25.247715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.65:46375)
0810 03:54:25.248046 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:25.248118 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":94,"threads_started":1}
I20260810 03:54:25.411159 26375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:25.410673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.66:54961)
0810 03:54:25.410986 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:25.411058 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":79,"threads_started":1}
I20260810 03:54:25.591678 26378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:25.591135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.67:57853)
0810 03:54:25.591490 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:25.591566 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":91,"threads_started":1}
I20260810 03:54:26.249725 26379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:26.249179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.65:55507)
0810 03:54:26.249532 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:26.249603 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":106,"threads_started":1}
I20260810 03:54:26.412432 26383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:26.412001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.66:57025)
0810 03:54:26.412267 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:26.412329 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":85,"threads_started":1}
I20260810 03:54:26.593030 26385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:26.592539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.67:45397)
0810 03:54:26.592863 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:26.592930 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":81,"threads_started":1}
I20260810 03:54:27.251098 26393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:27.250668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.65:38511)
0810 03:54:27.250942 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:27.250998 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":96,"threads_started":1}
I20260810 03:54:27.413918 26394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:27.413424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.66:41755)
0810 03:54:27.413744 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:27.413814 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":113,"threads_started":1}
I20260810 03:54:27.594485 26395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:27.593897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.67:48423)
0810 03:54:27.594261 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:27.594355 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":97,"threads_started":1}
I20260810 03:54:28.252421 26396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:28.251928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.65:37813)
0810 03:54:28.252222 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:28.252298 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":79,"threads_started":1}
I20260810 03:54:28.415213 26397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:28.414774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.66:58771)
0810 03:54:28.415054 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:28.415111 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":93,"threads_started":1}
I20260810 03:54:28.470507 26398 raft_consensus.cc:482] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:54:28.470670 26398 raft_consensus.cc:504] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } }
I20260810 03:54:28.471110 26398 leader_election.cc:288] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61990e39cfa44e9ca31271e185bc4787 (127.27.153.67:33387), caf97fd4c64e452aaa50b640477abb4a (127.27.153.66:34091)
W20260810 03:54:28.471913 25620 leader_election.cc:341] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61990e39cfa44e9ca31271e185bc4787 (127.27.153.67:33387): Not found: Tablet not found: 46fd6ebf839e495fae99efacbdf2d568
W20260810 03:54:28.472038 25615 leader_election.cc:341] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer caf97fd4c64e452aaa50b640477abb4a (127.27.153.66:34091): Not found: Tablet not found: 46fd6ebf839e495fae99efacbdf2d568
I20260810 03:54:28.472124 25615 leader_election.cc:302] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fcfc8fa8c7ac40dbb3ee8f78d7270547; no voters: 61990e39cfa44e9ca31271e185bc4787, caf97fd4c64e452aaa50b640477abb4a
I20260810 03:54:28.472288 26398 raft_consensus.cc:2726] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:54:28.595971 26399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:28.595451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.67:33513)
0810 03:54:28.595806 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:28.595870 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":125,"threads_started":1}
I20260810 03:54:29.253697 26400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:29.253253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.65:35431)
0810 03:54:29.253527 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:29.253592 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":78,"threads_started":1}
I20260810 03:54:29.416385 26402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:29.415992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.66:50683)
0810 03:54:29.416228 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:29.416285 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
I20260810 03:54:29.597370 26403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:29.596904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.67:34831)
0810 03:54:29.597197 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:29.597267 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":80,"threads_started":1}
I20260810 03:54:30.255201 26404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:30.254666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.65:57585)
0810 03:54:30.254992 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:30.255078 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":91,"threads_started":1}
I20260810 03:54:30.417650 26405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:30.417192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.66:46777)
0810 03:54:30.417473 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:30.417537 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":92,"threads_started":1}
I20260810 03:54:30.598785 26406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:30.598274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.67:60575)
0810 03:54:30.598598 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:30.598667 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":100,"threads_started":1}
I20260810 03:54:31.256661 26407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:31.256140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.65:50127)
0810 03:54:31.256467 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:31.256554 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":94,"threads_started":1}
I20260810 03:54:31.419114 26409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:31.418602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.66:56373)
0810 03:54:31.418930 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:31.418996 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":97,"threads_started":1}
I20260810 03:54:31.600265 26410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:31.599752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.67:39139)
0810 03:54:31.600064 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:31.600146 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":98,"threads_started":1}
I20260810 03:54:32.258157 26411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:32.257698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.65:54953)
0810 03:54:32.257988 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:32.258053 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":90,"threads_started":1}
I20260810 03:54:32.284907 26414 raft_consensus.cc:482] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:54:32.285044 26414 raft_consensus.cc:504] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61990e39cfa44e9ca31271e185bc4787" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 33387 } } peers { permanent_uuid: "fcfc8fa8c7ac40dbb3ee8f78d7270547" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 38975 } } peers { permanent_uuid: "caf97fd4c64e452aaa50b640477abb4a" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 34091 } }
I20260810 03:54:32.285483 26414 leader_election.cc:288] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61990e39cfa44e9ca31271e185bc4787 (127.27.153.67:33387), caf97fd4c64e452aaa50b640477abb4a (127.27.153.66:34091)
W20260810 03:54:32.286280 25615 leader_election.cc:341] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer caf97fd4c64e452aaa50b640477abb4a (127.27.153.66:34091): Not found: Tablet not found: 46fd6ebf839e495fae99efacbdf2d568
W20260810 03:54:32.286404 25620 leader_election.cc:341] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61990e39cfa44e9ca31271e185bc4787 (127.27.153.67:33387): Not found: Tablet not found: 46fd6ebf839e495fae99efacbdf2d568
I20260810 03:54:32.286505 25620 leader_election.cc:302] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fcfc8fa8c7ac40dbb3ee8f78d7270547; no voters: 61990e39cfa44e9ca31271e185bc4787, caf97fd4c64e452aaa50b640477abb4a
I20260810 03:54:32.286664 26414 raft_consensus.cc:2726] T 46fd6ebf839e495fae99efacbdf2d568 P fcfc8fa8c7ac40dbb3ee8f78d7270547 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 03:54:32.420629 26415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:32.420175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.66:45813)
0810 03:54:32.420456 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:32.420523 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":85,"threads_started":1}
I20260810 03:54:32.589988 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 25514
I20260810 03:54:32.603219 26416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:32.602760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42371 (local address 127.27.153.67:38781)
0810 03:54:32.603045 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:32.603108 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.126:42371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":88,"threads_started":1}
I20260810 03:54:32.608459 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 25844
I20260810 03:54:32.631273 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 26068
2026-08-10T03:54:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59728 ms)
[----------] 30 tests from SecurityITest (257768 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:54:32 dist-test-slave-pww0 krb5kdc[26424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:54:32 dist-test-slave-pww0 krb5kdc[26424](info): set up 2 sockets
Aug 10 03:54:32 dist-test-slave-pww0 krb5kdc[26424](info): commencing operation
krb5kdc: starting...
W20260810 03:54:33.847139 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:54:33 dist-test-slave-pww0 krb5kdc[26424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:54:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:54:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:54:33.977890 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:36039
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:40721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:36039
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:54:34.095896 26448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:34.096207 26448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:34.096279 26448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:34.100472 26448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:54:34.100554 26448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:34.100581 26448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:34.100602 26448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:54:34.100625 26448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:54:34.106302 26448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:36039
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:36039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:34.107537 26448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:34.108646 26448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:54:34.118157 26448 server_base.cc:1034] running on GCE node
W20260810 03:54:34.118110 26456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:34.118126 26459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:34.118100 26455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:34.119014 26448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:34.120263 26448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:34.121454 26448 hybrid_clock.cc:648] HybridClock initialized: now 1786334074121425 us; error 40 us; skew 500 ppm
Aug 10 03:54:34 dist-test-slave-pww0 krb5kdc[26424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334074, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:34.124323 26448 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:54:34.125375 26448 webserver.cc:460] Webserver started at http://127.27.153.126:42163/ using document root <none> and password file <none>
I20260810 03:54:34.125968 26448 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:34.126039 26448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:34.126252 26448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:34.128466 26448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "45fc2584a58a4c239dac17744846ebda"
format_stamp: "Formatted at 2026-08-10 03:54:34 on dist-test-slave-pww0"
server_key: "6535dd8e9bb6257fa1ae6c84210ea827"
server_key_iv: "3674e8845913bdcd2a097686a3371a1a"
server_key_version: "encryptionkey@0"
I20260810 03:54:34.129048 26448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "45fc2584a58a4c239dac17744846ebda"
format_stamp: "Formatted at 2026-08-10 03:54:34 on dist-test-slave-pww0"
server_key: "6535dd8e9bb6257fa1ae6c84210ea827"
server_key_iv: "3674e8845913bdcd2a097686a3371a1a"
server_key_version: "encryptionkey@0"
I20260810 03:54:34.132993 26448 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:54:34.135550 26475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:34.136845 26448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 03:54:34.136973 26448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "45fc2584a58a4c239dac17744846ebda"
format_stamp: "Formatted at 2026-08-10 03:54:34 on dist-test-slave-pww0"
server_key: "6535dd8e9bb6257fa1ae6c84210ea827"
server_key_iv: "3674e8845913bdcd2a097686a3371a1a"
server_key_version: "encryptionkey@0"
I20260810 03:54:34.137077 26448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:34.155894 26448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:34.159258 26448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:34.159494 26448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:34.166960 26448 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:36039
I20260810 03:54:34.166996 26569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:36039 every 8 connection(s)
I20260810 03:54:34.168008 26448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:54:34.170957 26571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:34.175920 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 26448
I20260810 03:54:34.176046 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:54:34.176329 28261 external_mini_cluster.cc:1428] Setting key 4f1ff7a4b19c0f558b8446ae0b24820d
I20260810 03:54:34.177074 26571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda: Bootstrap starting.
I20260810 03:54:34.180019 26571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:34.180955 26571 log.cc:824] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:34.183307 26571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda: No bootstrap required, opened a new log
Aug 10 03:54:34 dist-test-slave-pww0 krb5kdc[26424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:54:34.187584 26571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fc2584a58a4c239dac17744846ebda" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 36039 } }
I20260810 03:54:34.187922 26571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:34.187986 26571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45fc2584a58a4c239dac17744846ebda, State: Initialized, Role: FOLLOWER
I20260810 03:54:34.188534 26571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fc2584a58a4c239dac17744846ebda" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 36039 } }
I20260810 03:54:34.188702 26571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:54:34.188781 26571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:54:34.188887 26571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:34.190081 26571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fc2584a58a4c239dac17744846ebda" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 36039 } }
I20260810 03:54:34.190503 26571 leader_election.cc:302] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45fc2584a58a4c239dac17744846ebda; no voters:
I20260810 03:54:34.190862 26571 leader_election.cc:288] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:54:34.191010 26578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:54:34.191234 26578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda [term 1 LEADER]: Becoming Leader. State: Replica: 45fc2584a58a4c239dac17744846ebda, State: Running, Role: LEADER
I20260810 03:54:34.191646 26578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fc2584a58a4c239dac17744846ebda" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 36039 } }
I20260810 03:54:34.192118 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:34.177855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.27.153.126:36039)
0810 03:54:34.178319 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:34.178339 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:34.178388 (+ 49us) server_negotiation.cc:407] Connection header received
0810 03:54:34.179262 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:34.179288 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:34.179769 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:34.180169 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:34.180895 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.182226 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.183179 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.183909 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.184077 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:34.186569 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:34.186577 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:34.186598 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:34.186635 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:34.188578 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:34.189147 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:34.189154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:34.189159 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:34.189313 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:34.189603 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:34.189606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:34.189608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:34.190198 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:34.190394 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:34.190718 (+ 324us) server_negotiation.cc:299] Negotiation successful
0810 03:54:34.191049 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":127,"threads_started":1}
I20260810 03:54:34.192327 26571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:54:34.193374 26580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45fc2584a58a4c239dac17744846ebda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fc2584a58a4c239dac17744846ebda" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 36039 } } }
I20260810 03:54:34.193502 26580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:34.194478 26581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45fc2584a58a4c239dac17744846ebda. Latest consensus state: current_term: 1 leader_uuid: "45fc2584a58a4c239dac17744846ebda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45fc2584a58a4c239dac17744846ebda" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 36039 } } }
I20260810 03:54:34.194880 26581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:34.194938 26594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:54:34.197655 26594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:54:34.203789 26594 catalog_manager.cc:1347] Generated new cluster ID: b8d5a53400914b27bcd0e6cba59b1c83
I20260810 03:54:34.203884 26594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:54:34.211961 26594 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:54:34.213336 26594 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:54:34.220485 26594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda: Generated new TSK 0
I20260810 03:54:34.221380 26594 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.153.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
I20260810 03:54:34.254818 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:36039
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:40721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:54:34.373131 26616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:34.373420 26616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:34.373464 26616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:34.378100 26616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:34.378183 26616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:34.378250 26616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:54:34.384228 26616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:36039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:34.385351 26616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:34.386401 26616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:34.396651 26634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:34.396654 26628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:34.396651 26631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:34.396826 26616 server_base.cc:1034] running on GCE node
I20260810 03:54:34.397399 26616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:34.398090 26616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:34.399266 26616 hybrid_clock.cc:648] HybridClock initialized: now 1786334074399243 us; error 38 us; skew 500 ppm
Aug 10 03:54:34 dist-test-slave-pww0 krb5kdc[26424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334074, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:34.402179 26616 init.cc:377] Logged in from keytab as kudu/127.27.153.65@KRBTEST.COM (short username kudu)
I20260810 03:54:34.403170 26616 webserver.cc:460] Webserver started at http://127.27.153.65:45005/ using document root <none> and password file <none>
I20260810 03:54:34.403797 26616 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:34.403867 26616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:34.404073 26616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:34.406208 26616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "b89b32e3545645efb377d9516935fd86"
format_stamp: "Formatted at 2026-08-10 03:54:34 on dist-test-slave-pww0"
server_key: "67af9a769b11aeaa2ba40bb0d4b2b376"
server_key_iv: "e9073a3ff54b8b72c09a98d6786bbf0b"
server_key_version: "encryptionkey@0"
I20260810 03:54:34.406765 26616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "b89b32e3545645efb377d9516935fd86"
format_stamp: "Formatted at 2026-08-10 03:54:34 on dist-test-slave-pww0"
server_key: "67af9a769b11aeaa2ba40bb0d4b2b376"
server_key_iv: "e9073a3ff54b8b72c09a98d6786bbf0b"
server_key_version: "encryptionkey@0"
I20260810 03:54:34.410818 26616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:54:34.413968 26646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:34.415634 26616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:54:34.415798 26616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "b89b32e3545645efb377d9516935fd86"
format_stamp: "Formatted at 2026-08-10 03:54:34 on dist-test-slave-pww0"
server_key: "67af9a769b11aeaa2ba40bb0d4b2b376"
server_key_iv: "e9073a3ff54b8b72c09a98d6786bbf0b"
server_key_version: "encryptionkey@0"
I20260810 03:54:34.416002 26616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:34.440353 26616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:34.443550 26616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:34.443718 26616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:34.444298 26616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:54:34.445139 26616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:54:34.445187 26616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:34.445228 26616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:54:34.445247 26616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:34.454080 26616 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:46055
I20260810 03:54:34.454110 26856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:46055 every 8 connection(s)
I20260810 03:54:34.455045 26616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:54:34.455935 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 26616
I20260810 03:54:34.456036 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:54:34.456316 28261 external_mini_cluster.cc:1428] Setting key 4d85b05cb13b8480018e219afe98995c
Aug 10 03:54:34 dist-test-slave-pww0 krb5kdc[26424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334074, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.153.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.66@KRBTEST.COM" created.
I20260810 03:54:34.469836 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:34.457168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:57605 (local address 127.27.153.126:36039)
0810 03:54:34.457310 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:34.457314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:34.458253 (+ 939us) server_negotiation.cc:407] Connection header received
0810 03:54:34.459673 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:34.459677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:34.459736 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:34.459813 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:34.461056 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.461613 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.462860 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.463100 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.463193 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:34.465776 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:34.465780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:34.465782 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:34.465812 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:34.467280 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:34.467987 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:34.467991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:34.467992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:34.468081 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:34.468531 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:34.468534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:34.468535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:34.468739 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:34.468829 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:34.469513 (+ 684us) server_negotiation.cc:299] Negotiation successful
0810 03:54:34.469657 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:54:34.470767 26861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:34.457530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:36039 (local address 127.27.153.65:57605)
0810 03:54:34.458098 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:34.458145 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:34.459415 (+ 1270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:34.460020 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:34.460030 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:34.460404 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:54:34.460908 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:34.460922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.461745 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.461749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:34.462741 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:34.462753 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.463257 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.463262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:34.463438 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:34.464057 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:54:34.464075 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:54:34.465579 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:54:34.467415 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:54:34.467423 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:54:34.467442 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:54:34.467851 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:54:34.468176 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:54:34.468179 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:54:34.468182 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:54:34.468420 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:54:34.468843 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:54:34.468851 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:54:34.469225 (+ 374us) client_negotiation.cc:769] Sending connection context
0810 03:54:34.469449 (+ 224us) client_negotiation.cc:240] Negotiation successful
0810 03:54:34.469840 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":164,"threads_started":1}
I20260810 03:54:34.472134 26858 heartbeater.cc:347] Connected to a master server at 127.27.153.126:36039
I20260810 03:54:34.472440 26858 heartbeater.cc:464] Registering TS with master...
I20260810 03:54:34.472985 26858 heartbeater.cc:511] Master 127.27.153.126:36039 requested a full tablet report, sending...
I20260810 03:54:34.474735 26509 ts_manager.cc:194] Registered new tserver with Master: b89b32e3545645efb377d9516935fd86 (127.27.153.65:46055)
I20260810 03:54:34.476528 26509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:57605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
I20260810 03:54:34.488080 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:36039
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:40721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:54:34.605341 26866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:34.605669 26866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:34.605746 26866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:34.609910 26866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:34.610006 26866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:34.610112 26866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:54:34.615944 26866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:36039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:34.617132 26866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:34.618204 26866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:34.628320 26875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:34.628499 26866 server_base.cc:1034] running on GCE node
W20260810 03:54:34.628324 26878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:34.628320 26880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:34.629150 26866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:34.629896 26866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:34.631098 26866 hybrid_clock.cc:648] HybridClock initialized: now 1786334074631074 us; error 42 us; skew 500 ppm
Aug 10 03:54:34 dist-test-slave-pww0 krb5kdc[26424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334074, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:34.634013 26866 init.cc:377] Logged in from keytab as kudu/127.27.153.66@KRBTEST.COM (short username kudu)
I20260810 03:54:34.635017 26866 webserver.cc:460] Webserver started at http://127.27.153.66:39585/ using document root <none> and password file <none>
I20260810 03:54:34.635646 26866 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:34.635720 26866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:34.635931 26866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:34.638085 26866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "d9067b5d6a3543f0b9a6e1aaffe52518"
format_stamp: "Formatted at 2026-08-10 03:54:34 on dist-test-slave-pww0"
server_key: "19c876732975b6265803f542c9b8caae"
server_key_iv: "b01b003fac74b3ada346ed4c535c297a"
server_key_version: "encryptionkey@0"
I20260810 03:54:34.638847 26866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "d9067b5d6a3543f0b9a6e1aaffe52518"
format_stamp: "Formatted at 2026-08-10 03:54:34 on dist-test-slave-pww0"
server_key: "19c876732975b6265803f542c9b8caae"
server_key_iv: "b01b003fac74b3ada346ed4c535c297a"
server_key_version: "encryptionkey@0"
I20260810 03:54:34.643266 26866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260810 03:54:34.646533 26890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:34.647995 26866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:54:34.648132 26866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "d9067b5d6a3543f0b9a6e1aaffe52518"
format_stamp: "Formatted at 2026-08-10 03:54:34 on dist-test-slave-pww0"
server_key: "19c876732975b6265803f542c9b8caae"
server_key_iv: "b01b003fac74b3ada346ed4c535c297a"
server_key_version: "encryptionkey@0"
I20260810 03:54:34.648269 26866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:34.686820 26866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:34.689985 26866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:34.690181 26866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:34.690852 26866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:54:34.691879 26866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:54:34.691931 26866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:34.691973 26866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:54:34.691993 26866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:34.701290 26866 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:37569
I20260810 03:54:34.701320 27102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:37569 every 8 connection(s)
I20260810 03:54:34.702208 26866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:54:34.706712 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 26866
I20260810 03:54:34.706859 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:54:34.707140 28261 external_mini_cluster.cc:1428] Setting key 33e25c59035f9c0c7229df68e392e084
Aug 10 03:54:34 dist-test-slave-pww0 krb5kdc[26424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334074, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:54:34.717082 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:34.704062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:49623 (local address 127.27.153.126:36039)
0810 03:54:34.704209 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:34.704213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:34.705122 (+ 909us) server_negotiation.cc:407] Connection header received
0810 03:54:34.706552 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:34.706556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:34.706620 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:34.706750 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:34.708142 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.708788 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.710030 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.710343 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.710446 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:34.713124 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:34.713128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:34.713131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:34.713164 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:34.714792 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:34.715330 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:34.715334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:34.715336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:34.715438 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:34.715838 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:34.715841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:34.715843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:34.716099 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:34.716204 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:34.716761 (+ 557us) server_negotiation.cc:299] Negotiation successful
0810 03:54:34.716902 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 03:54:34.717928 27108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:34.704488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:36039 (local address 127.27.153.66:49623)
0810 03:54:34.704971 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:34.705015 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:34.706306 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:34.706937 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:34.706947 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:34.707351 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:54:34.707968 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:34.707991 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.708940 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.708943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:34.709896 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:34.709908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.710492 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.710495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:34.710655 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:34.711295 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:54:34.711313 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:54:34.712925 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:54:34.714918 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:54:34.714924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:54:34.714939 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:54:34.715219 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:54:34.715564 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:54:34.715567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:54:34.715569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:54:34.715726 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:54:34.716216 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:54:34.716222 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:54:34.716503 (+ 281us) client_negotiation.cc:769] Sending connection context
0810 03:54:34.716671 (+ 168us) client_negotiation.cc:240] Negotiation successful
0810 03:54:34.716988 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":127,"threads_started":1}
WARNING: no policy specified for kudu/127.27.153.67@KRBTEST.COM; defaulting to no policy
I20260810 03:54:34.719184 27106 heartbeater.cc:347] Connected to a master server at 127.27.153.126:36039
I20260810 03:54:34.719569 27106 heartbeater.cc:464] Registering TS with master...
I20260810 03:54:34.720181 27106 heartbeater.cc:511] Master 127.27.153.126:36039 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.67@KRBTEST.COM" created.
I20260810 03:54:34.721645 26509 ts_manager.cc:194] Registered new tserver with Master: d9067b5d6a3543f0b9a6e1aaffe52518 (127.27.153.66:37569)
I20260810 03:54:34.722371 26509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:49623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
I20260810 03:54:34.738615 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:36039
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:40721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:54:34.853668 27113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:34.853971 27113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:34.854034 27113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:34.858278 27113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:34.858362 27113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:34.858448 27113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:54:34.864171 27113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:36039
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:34.865376 27113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:34.866431 27113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:34.876348 27125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:34.876331 27120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:34.876335 27122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:34.876727 27113 server_base.cc:1034] running on GCE node
I20260810 03:54:34.877147 27113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:34.877815 27113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:34.879011 27113 hybrid_clock.cc:648] HybridClock initialized: now 1786334074878992 us; error 29 us; skew 500 ppm
Aug 10 03:54:34 dist-test-slave-pww0 krb5kdc[26424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334074, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:34.881973 27113 init.cc:377] Logged in from keytab as kudu/127.27.153.67@KRBTEST.COM (short username kudu)
I20260810 03:54:34.883010 27113 webserver.cc:460] Webserver started at http://127.27.153.67:38723/ using document root <none> and password file <none>
I20260810 03:54:34.883659 27113 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:34.883740 27113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:34.883967 27113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:34.886206 27113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "d9c3c990e7384930a3a720c3af4458aa"
format_stamp: "Formatted at 2026-08-10 03:54:34 on dist-test-slave-pww0"
server_key: "5d4a41e0e6527dbb3c4c9bcc24801f30"
server_key_iv: "356fe101102886a1fee0c7f9e27fca5e"
server_key_version: "encryptionkey@0"
I20260810 03:54:34.886823 27113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "d9c3c990e7384930a3a720c3af4458aa"
format_stamp: "Formatted at 2026-08-10 03:54:34 on dist-test-slave-pww0"
server_key: "5d4a41e0e6527dbb3c4c9bcc24801f30"
server_key_iv: "356fe101102886a1fee0c7f9e27fca5e"
server_key_version: "encryptionkey@0"
I20260810 03:54:34.890978 27113 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260810 03:54:34.893761 27137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:34.895277 27113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:54:34.895414 27113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "d9c3c990e7384930a3a720c3af4458aa"
format_stamp: "Formatted at 2026-08-10 03:54:34 on dist-test-slave-pww0"
server_key: "5d4a41e0e6527dbb3c4c9bcc24801f30"
server_key_iv: "356fe101102886a1fee0c7f9e27fca5e"
server_key_version: "encryptionkey@0"
I20260810 03:54:34.895574 27113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:34.912895 27113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:34.916260 27113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:34.916429 27113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:34.917038 27113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:54:34.917929 27113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:54:34.917979 27113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:34.918035 27113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:54:34.918061 27113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:34.927042 27113 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:39555
I20260810 03:54:34.927074 27348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:39555 every 8 connection(s)
I20260810 03:54:34.928032 27113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:54:34.936715 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 27113
I20260810 03:54:34.936859 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:54:34.937134 28261 external_mini_cluster.cc:1428] Setting key 77606bcacc7857911666b1e60eaa351a
Aug 10 03:54:34 dist-test-slave-pww0 krb5kdc[26424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334074, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:54:34.942139 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:34.929930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:47721 (local address 127.27.153.126:36039)
0810 03:54:34.930074 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:34.930078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:34.930977 (+ 899us) server_negotiation.cc:407] Connection header received
0810 03:54:34.932328 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:34.932331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:34.932382 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:34.932450 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:34.933650 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.934192 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.935380 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.935698 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.935793 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:34.938542 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:34.938546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:34.938548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:34.938577 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:34.940002 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:34.940559 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:34.940563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:34.940565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:34.940646 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:34.940982 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:34.940985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:34.940987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:34.941185 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:34.941274 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:34.941841 (+ 567us) server_negotiation.cc:299] Negotiation successful
0810 03:54:34.941970 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 03:54:34.943027 27353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:34.930329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:36039 (local address 127.27.153.67:47721)
0810 03:54:34.930816 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:34.930862 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:34.932103 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:34.932606 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:34.932613 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:34.932999 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:54:34.933505 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:34.933519 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.934311 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.934314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:34.935235 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:34.935246 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.935849 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.935852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:34.936010 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:34.936667 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:54:34.936686 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:54:34.938364 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:54:34.940135 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:54:34.940141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:54:34.940160 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:54:34.940444 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:54:34.940734 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:54:34.940738 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:54:34.940740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:54:34.940892 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:54:34.941294 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:54:34.941303 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:54:34.941581 (+ 278us) client_negotiation.cc:769] Sending connection context
0810 03:54:34.941779 (+ 198us) client_negotiation.cc:240] Negotiation successful
0810 03:54:34.942140 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":121,"threads_started":1}
I20260810 03:54:34.944191 27349 heartbeater.cc:347] Connected to a master server at 127.27.153.126:36039
I20260810 03:54:34.944474 27349 heartbeater.cc:464] Registering TS with master...
I20260810 03:54:34.945017 27349 heartbeater.cc:511] Master 127.27.153.126:36039 requested a full tablet report, sending...
I20260810 03:54:34.946285 26509 ts_manager.cc:194] Registered new tserver with Master: d9c3c990e7384930a3a720c3af4458aa (127.27.153.67:39555)
I20260810 03:54:34.946892 26509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:47721
I20260810 03:54:34.951841 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:54:34.961997 26574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:34.954009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53966 (local address 127.27.153.126:36039)
0810 03:54:34.954159 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:34.954162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:34.954289 (+ 127us) server_negotiation.cc:407] Connection header received
0810 03:54:34.954421 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:34.954423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:34.954468 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:34.954529 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:34.955312 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.955892 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.956823 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.957050 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.957135 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:34.958603 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:34.958606 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:34.958608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:34.958642 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:34.960158 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:34.960687 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:34.960690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:34.960691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:34.960785 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:34.961067 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:34.961070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:34.961071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:34.961335 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:34.961446 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:34.961752 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 03:54:34.961870 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:54:34.965302 26509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:54:34.968665 26509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:54:34.983690 27372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:34.978331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:46055 (local address 127.0.0.1:43424)
0810 03:54:34.979091 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:34.979104 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:34.979188 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:34.979793 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:34.979796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:34.979813 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:34.979915 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:34.979922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.980942 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.980946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:34.982133 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:34.982142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.983300 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.983304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:34.983392 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:34.983413 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 03:54:34.983507 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 03:54:34.983562 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":1,"thread_start_us":94,"threads_started":1}
I20260810 03:54:34.983807 27379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:34.978677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43424 (local address 127.27.153.65:46055)
0810 03:54:34.979411 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:34.979417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:34.979435 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:54:34.979522 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:34.979527 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:34.979670 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:34.979792 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:34.980042 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.980828 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.982281 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.983091 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.983193 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:34.983342 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:34.983545 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 03:54:34.983652 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":100,"threads_started":1}
I20260810 03:54:34.984002 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:34.978113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.27.153.66:37569)
0810 03:54:34.978464 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:34.978478 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:34.978505 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:54:34.978582 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:34.978587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:34.978810 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:34.979375 (+ 565us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:34.979577 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.980710 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.982140 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.982964 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.983056 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:34.983213 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:34.983755 (+ 542us) server_negotiation.cc:299] Negotiation successful
0810 03:54:34.983873 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":166,"threads_started":1}
I20260810 03:54:34.984205 27370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:34.977685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:39555 (local address 127.0.0.1:54094)
0810 03:54:34.978990 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:34.979003 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:34.979076 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:34.979984 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:34.979987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:34.980005 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:34.980104 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:34.980111 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.981447 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.981451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:34.982676 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:34.982688 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.983825 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.983829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:34.983929 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:34.983954 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 03:54:34.984004 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 03:54:34.984074 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1240,"mutex_wait_us":39,"thread_start_us":91,"threads_started":1}
I20260810 03:54:34.983690 27369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:34.977583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:37569 (local address 127.0.0.1:47650)
0810 03:54:34.977948 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:34.978009 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:34.978147 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:34.978991 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:34.978997 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:34.979025 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:34.979194 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:34.979201 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.980853 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.980856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:34.981998 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:34.982010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.983070 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.983073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:34.983191 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:34.983232 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 03:54:34.983339 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 03:54:34.983512 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":53,"thread_start_us":90,"threads_started":1}
I20260810 03:54:34.984694 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:34.978564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54094 (local address 127.27.153.67:39555)
0810 03:54:34.979481 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:34.979493 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:34.979518 (+ 25us) server_negotiation.cc:407] Connection header received
0810 03:54:34.979592 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:34.979597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:34.979800 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:34.979985 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:34.980214 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.981272 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.982816 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:34.983672 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:34.984298 (+ 626us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:34.984414 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:34.984465 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 03:54:34.984556 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":790,"thread_start_us":163,"threads_started":1}
I20260810 03:54:34.987315 26747 tablet_service.cc:1467] Processing CreateTablet for tablet 4a7949c81c464c29929724f9543d9610 (DEFAULT_TABLE table=test-table [id=dc6b11d66b214a739ad9afab4619f8e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:54:34.987423 26977 tablet_service.cc:1467] Processing CreateTablet for tablet 4a7949c81c464c29929724f9543d9610 (DEFAULT_TABLE table=test-table [id=dc6b11d66b214a739ad9afab4619f8e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:54:34.987423 27228 tablet_service.cc:1467] Processing CreateTablet for tablet 4a7949c81c464c29929724f9543d9610 (DEFAULT_TABLE table=test-table [id=dc6b11d66b214a739ad9afab4619f8e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:54:34.988397 26747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a7949c81c464c29929724f9543d9610. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:34.988479 26977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a7949c81c464c29929724f9543d9610. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:34.988479 27228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a7949c81c464c29929724f9543d9610. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:34.995720 27381 tablet_bootstrap.cc:492] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518: Bootstrap starting.
I20260810 03:54:34.996456 27380 tablet_bootstrap.cc:492] T 4a7949c81c464c29929724f9543d9610 P b89b32e3545645efb377d9516935fd86: Bootstrap starting.
I20260810 03:54:34.996573 27382 tablet_bootstrap.cc:492] T 4a7949c81c464c29929724f9543d9610 P d9c3c990e7384930a3a720c3af4458aa: Bootstrap starting.
I20260810 03:54:34.997972 27381 tablet_bootstrap.cc:654] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:34.998661 27380 tablet_bootstrap.cc:654] T 4a7949c81c464c29929724f9543d9610 P b89b32e3545645efb377d9516935fd86: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:34.998661 27382 tablet_bootstrap.cc:654] T 4a7949c81c464c29929724f9543d9610 P d9c3c990e7384930a3a720c3af4458aa: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:34.999177 27381 log.cc:824] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:34.999614 27380 log.cc:824] T 4a7949c81c464c29929724f9543d9610 P b89b32e3545645efb377d9516935fd86: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:35.000000 27382 log.cc:824] T 4a7949c81c464c29929724f9543d9610 P d9c3c990e7384930a3a720c3af4458aa: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:35.001842 27382 tablet_bootstrap.cc:492] T 4a7949c81c464c29929724f9543d9610 P d9c3c990e7384930a3a720c3af4458aa: No bootstrap required, opened a new log
I20260810 03:54:35.001853 27381 tablet_bootstrap.cc:492] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518: No bootstrap required, opened a new log
I20260810 03:54:35.002053 27382 ts_tablet_manager.cc:1397] T 4a7949c81c464c29929724f9543d9610 P d9c3c990e7384930a3a720c3af4458aa: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:54:35.002055 27381 ts_tablet_manager.cc:1397] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 03:54:35.002421 27380 tablet_bootstrap.cc:492] T 4a7949c81c464c29929724f9543d9610 P b89b32e3545645efb377d9516935fd86: No bootstrap required, opened a new log
I20260810 03:54:35.002625 27380 ts_tablet_manager.cc:1397] T 4a7949c81c464c29929724f9543d9610 P b89b32e3545645efb377d9516935fd86: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:54:35.005223 27382 raft_consensus.cc:348] T 4a7949c81c464c29929724f9543d9610 P d9c3c990e7384930a3a720c3af4458aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9067b5d6a3543f0b9a6e1aaffe52518" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 37569 } } peers { permanent_uuid: "d9c3c990e7384930a3a720c3af4458aa" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 39555 } } peers { permanent_uuid: "b89b32e3545645efb377d9516935fd86" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 46055 } }
I20260810 03:54:35.005484 27382 raft_consensus.cc:374] T 4a7949c81c464c29929724f9543d9610 P d9c3c990e7384930a3a720c3af4458aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:35.005548 27382 raft_consensus.cc:729] T 4a7949c81c464c29929724f9543d9610 P d9c3c990e7384930a3a720c3af4458aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9c3c990e7384930a3a720c3af4458aa, State: Initialized, Role: FOLLOWER
I20260810 03:54:35.006011 27382 consensus_queue.cc:260] T 4a7949c81c464c29929724f9543d9610 P d9c3c990e7384930a3a720c3af4458aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9067b5d6a3543f0b9a6e1aaffe52518" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 37569 } } peers { permanent_uuid: "d9c3c990e7384930a3a720c3af4458aa" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 39555 } } peers { permanent_uuid: "b89b32e3545645efb377d9516935fd86" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 46055 } }
I20260810 03:54:35.006367 27381 raft_consensus.cc:348] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9067b5d6a3543f0b9a6e1aaffe52518" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 37569 } } peers { permanent_uuid: "d9c3c990e7384930a3a720c3af4458aa" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 39555 } } peers { permanent_uuid: "b89b32e3545645efb377d9516935fd86" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 46055 } }
I20260810 03:54:35.006682 27381 raft_consensus.cc:374] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:35.006745 27381 raft_consensus.cc:729] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9067b5d6a3543f0b9a6e1aaffe52518, State: Initialized, Role: FOLLOWER
I20260810 03:54:35.006732 27349 heartbeater.cc:503] Master 127.27.153.126:36039 was elected leader, sending a full tablet report...
I20260810 03:54:35.007131 27382 ts_tablet_manager.cc:1428] T 4a7949c81c464c29929724f9543d9610 P d9c3c990e7384930a3a720c3af4458aa: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 03:54:35.007253 27381 consensus_queue.cc:260] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9067b5d6a3543f0b9a6e1aaffe52518" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 37569 } } peers { permanent_uuid: "d9c3c990e7384930a3a720c3af4458aa" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 39555 } } peers { permanent_uuid: "b89b32e3545645efb377d9516935fd86" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 46055 } }
I20260810 03:54:35.008365 27381 ts_tablet_manager.cc:1428] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 03:54:35.008189 27380 raft_consensus.cc:348] T 4a7949c81c464c29929724f9543d9610 P b89b32e3545645efb377d9516935fd86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9067b5d6a3543f0b9a6e1aaffe52518" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 37569 } } peers { permanent_uuid: "d9c3c990e7384930a3a720c3af4458aa" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 39555 } } peers { permanent_uuid: "b89b32e3545645efb377d9516935fd86" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 46055 } }
I20260810 03:54:35.008493 27380 raft_consensus.cc:374] T 4a7949c81c464c29929724f9543d9610 P b89b32e3545645efb377d9516935fd86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:35.008540 27380 raft_consensus.cc:729] T 4a7949c81c464c29929724f9543d9610 P b89b32e3545645efb377d9516935fd86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b89b32e3545645efb377d9516935fd86, State: Initialized, Role: FOLLOWER
I20260810 03:54:35.008677 27106 heartbeater.cc:503] Master 127.27.153.126:36039 was elected leader, sending a full tablet report...
I20260810 03:54:35.008950 27380 consensus_queue.cc:260] T 4a7949c81c464c29929724f9543d9610 P b89b32e3545645efb377d9516935fd86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9067b5d6a3543f0b9a6e1aaffe52518" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 37569 } } peers { permanent_uuid: "d9c3c990e7384930a3a720c3af4458aa" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 39555 } } peers { permanent_uuid: "b89b32e3545645efb377d9516935fd86" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 46055 } }
I20260810 03:54:35.009578 26858 heartbeater.cc:503] Master 127.27.153.126:36039 was elected leader, sending a full tablet report...
I20260810 03:54:35.009907 27380 ts_tablet_manager.cc:1428] T 4a7949c81c464c29929724f9543d9610 P b89b32e3545645efb377d9516935fd86: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 03:54:35.019548 27396 raft_consensus.cc:482] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:54:35.019793 27396 raft_consensus.cc:504] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9067b5d6a3543f0b9a6e1aaffe52518" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 37569 } } peers { permanent_uuid: "d9c3c990e7384930a3a720c3af4458aa" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 39555 } } peers { permanent_uuid: "b89b32e3545645efb377d9516935fd86" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 46055 } }
I20260810 03:54:35.020958 27396 leader_election.cc:288] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9c3c990e7384930a3a720c3af4458aa (127.27.153.67:39555), b89b32e3545645efb377d9516935fd86 (127.27.153.65:46055)
I20260810 03:54:35.025346 27108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:35.021235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:39555 (local address 127.27.153.66:58813)
0810 03:54:35.021358 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:35.021374 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:35.021473 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:35.021737 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:35.021745 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:35.021765 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:35.021912 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:35.021919 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:35.022807 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:35.022810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:35.024075 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:35.024086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:35.024923 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:35.024927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:35.025029 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:35.025051 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 03:54:35.025117 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 03:54:35.025198 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":40}
I20260810 03:54:35.025346 27378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:35.021343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:58813 (local address 127.27.153.67:39555)
0810 03:54:35.021467 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:35.021472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:35.021488 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:54:35.021570 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:35.021573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:35.021626 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:35.021718 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:35.022025 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:35.022687 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:35.024212 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:35.024805 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:35.024937 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:35.024995 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:35.025152 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 03:54:35.025229 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 03:54:35.025547 27379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:35.021343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:56475 (local address 127.27.153.65:46055)
0810 03:54:35.021467 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:35.021472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:35.021807 (+ 335us) server_negotiation.cc:407] Connection header received
0810 03:54:35.021942 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:35.021946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:35.021992 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:35.022053 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:35.022343 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:35.023018 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:35.024422 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:35.025022 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:35.025130 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:35.025176 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:35.025347 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 03:54:35.025442 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 03:54:35.025543 27399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:35.021235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:46055 (local address 127.27.153.66:56475)
0810 03:54:35.021717 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:35.021730 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:35.021828 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:35.022069 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:35.022072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:35.022087 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:35.022205 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:35.022212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:35.023120 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:35.023123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:35.024314 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:35.024322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:35.025132 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:35.025136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:35.025233 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:35.025259 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 03:54:35.025323 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 03:54:35.025395 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":137,"thread_start_us":135,"threads_started":1}
I20260810 03:54:35.026340 27265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7949c81c464c29929724f9543d9610" candidate_uuid: "d9067b5d6a3543f0b9a6e1aaffe52518" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9c3c990e7384930a3a720c3af4458aa" is_pre_election: true
I20260810 03:54:35.026372 26777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7949c81c464c29929724f9543d9610" candidate_uuid: "d9067b5d6a3543f0b9a6e1aaffe52518" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89b32e3545645efb377d9516935fd86" is_pre_election: true
I20260810 03:54:35.026677 26777 raft_consensus.cc:2454] T 4a7949c81c464c29929724f9543d9610 P b89b32e3545645efb377d9516935fd86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9067b5d6a3543f0b9a6e1aaffe52518 in term 0.
I20260810 03:54:35.026677 27265 raft_consensus.cc:2454] T 4a7949c81c464c29929724f9543d9610 P d9c3c990e7384930a3a720c3af4458aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9067b5d6a3543f0b9a6e1aaffe52518 in term 0.
I20260810 03:54:35.027254 26895 leader_election.cc:302] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b89b32e3545645efb377d9516935fd86, d9067b5d6a3543f0b9a6e1aaffe52518, d9c3c990e7384930a3a720c3af4458aa; no voters:
I20260810 03:54:35.027601 27396 raft_consensus.cc:2781] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:54:35.027683 27396 raft_consensus.cc:482] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:54:35.027712 27396 raft_consensus.cc:3037] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:35.028792 27396 raft_consensus.cc:504] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9067b5d6a3543f0b9a6e1aaffe52518" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 37569 } } peers { permanent_uuid: "d9c3c990e7384930a3a720c3af4458aa" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 39555 } } peers { permanent_uuid: "b89b32e3545645efb377d9516935fd86" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 46055 } }
I20260810 03:54:35.029152 27396 leader_election.cc:288] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518 [CANDIDATE]: Term 1 election: Requested vote from peers d9c3c990e7384930a3a720c3af4458aa (127.27.153.67:39555), b89b32e3545645efb377d9516935fd86 (127.27.153.65:46055)
I20260810 03:54:35.029604 26777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7949c81c464c29929724f9543d9610" candidate_uuid: "d9067b5d6a3543f0b9a6e1aaffe52518" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b89b32e3545645efb377d9516935fd86"
I20260810 03:54:35.029752 26777 raft_consensus.cc:3037] T 4a7949c81c464c29929724f9543d9610 P b89b32e3545645efb377d9516935fd86 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:35.030923 26777 raft_consensus.cc:2454] T 4a7949c81c464c29929724f9543d9610 P b89b32e3545645efb377d9516935fd86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9067b5d6a3543f0b9a6e1aaffe52518 in term 1.
I20260810 03:54:35.031275 27265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a7949c81c464c29929724f9543d9610" candidate_uuid: "d9067b5d6a3543f0b9a6e1aaffe52518" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9c3c990e7384930a3a720c3af4458aa"
I20260810 03:54:35.031327 26897 leader_election.cc:302] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b89b32e3545645efb377d9516935fd86, d9067b5d6a3543f0b9a6e1aaffe52518; no voters:
I20260810 03:54:35.031424 27265 raft_consensus.cc:3037] T 4a7949c81c464c29929724f9543d9610 P d9c3c990e7384930a3a720c3af4458aa [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:35.031520 27396 raft_consensus.cc:2781] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:54:35.031796 27396 raft_consensus.cc:686] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518 [term 1 LEADER]: Becoming Leader. State: Replica: d9067b5d6a3543f0b9a6e1aaffe52518, State: Running, Role: LEADER
I20260810 03:54:35.032200 27396 consensus_queue.cc:237] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9067b5d6a3543f0b9a6e1aaffe52518" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 37569 } } peers { permanent_uuid: "d9c3c990e7384930a3a720c3af4458aa" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 39555 } } peers { permanent_uuid: "b89b32e3545645efb377d9516935fd86" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 46055 } }
I20260810 03:54:35.032909 27265 raft_consensus.cc:2454] T 4a7949c81c464c29929724f9543d9610 P d9c3c990e7384930a3a720c3af4458aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9067b5d6a3543f0b9a6e1aaffe52518 in term 1.
I20260810 03:54:35.035570 26500 catalog_manager.cc:5495] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9067b5d6a3543f0b9a6e1aaffe52518 (127.27.153.66). New cstate: current_term: 1 leader_uuid: "d9067b5d6a3543f0b9a6e1aaffe52518" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9067b5d6a3543f0b9a6e1aaffe52518" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 37569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9c3c990e7384930a3a720c3af4458aa" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 39555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b89b32e3545645efb377d9516935fd86" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 46055 } health_report { overall_health: UNKNOWN } } }
I20260810 03:54:35.048070 27371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:35.044653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47654 (local address 127.27.153.66:37569)
0810 03:54:35.044787 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:35.044791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:35.044804 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:54:35.044924 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:35.044926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:35.044982 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:35.045069 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:54:35.045257 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:35.045771 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:35.046605 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:35.046826 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:35.046898 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:35.047175 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:54:35.047660 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:54:35.047769 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:35.047833 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 03:54:35.047891 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 03:54:35.054410 26777 raft_consensus.cc:1261] T 4a7949c81c464c29929724f9543d9610 P b89b32e3545645efb377d9516935fd86 [term 1 FOLLOWER]: Refusing update from remote peer d9067b5d6a3543f0b9a6e1aaffe52518: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:54:35.055124 27265 raft_consensus.cc:1261] T 4a7949c81c464c29929724f9543d9610 P d9c3c990e7384930a3a720c3af4458aa [term 1 FOLLOWER]: Refusing update from remote peer d9067b5d6a3543f0b9a6e1aaffe52518: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:54:35.055155 27402 consensus_queue.cc:1035] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b89b32e3545645efb377d9516935fd86" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 46055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:54:35.055792 27402 consensus_queue.cc:1035] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9c3c990e7384930a3a720c3af4458aa" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 39555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:54:35.062853 27406 mvcc.cc:205] Tried to move back new op lower bound from 7316824371414966272 to 7316824371335626752. Current Snapshot: MvccSnapshot[applied={T|T < 7316824371414966272}]
I20260810 03:54:35.063508 27412 mvcc.cc:205] Tried to move back new op lower bound from 7316824371414966272 to 7316824371335626752. Current Snapshot: MvccSnapshot[applied={T|T < 7316824371414966272}]
I20260810 03:54:35.065888 27407 mvcc.cc:205] Tried to move back new op lower bound from 7316824371414966272 to 7316824371335626752. Current Snapshot: MvccSnapshot[applied={T|T < 7316824371414966272}]
I20260810 03:54:35.073261 26500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:54:35.073434 26500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:54:35.075919 26500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45fc2584a58a4c239dac17744846ebda: Sending DeleteTablet for 3 replicas of tablet 4a7949c81c464c29929724f9543d9610
I20260810 03:54:35.076910 26747 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a7949c81c464c29929724f9543d9610 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:54:35 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:43424
I20260810 03:54:35.076877 27228 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a7949c81c464c29929724f9543d9610 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:54:35 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:54094
I20260810 03:54:35.077044 26977 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a7949c81c464c29929724f9543d9610 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:54:35 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:47650
I20260810 03:54:35.077445 27426 tablet_replica.cc:330] T 4a7949c81c464c29929724f9543d9610 P b89b32e3545645efb377d9516935fd86: stopping tablet replica
I20260810 03:54:35.077593 27424 tablet_replica.cc:330] T 4a7949c81c464c29929724f9543d9610 P d9c3c990e7384930a3a720c3af4458aa: stopping tablet replica
I20260810 03:54:35.077813 27426 raft_consensus.cc:2229] T 4a7949c81c464c29929724f9543d9610 P b89b32e3545645efb377d9516935fd86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:54:35.077896 27424 raft_consensus.cc:2229] T 4a7949c81c464c29929724f9543d9610 P d9c3c990e7384930a3a720c3af4458aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:54:35.078178 27426 raft_consensus.cc:2258] T 4a7949c81c464c29929724f9543d9610 P b89b32e3545645efb377d9516935fd86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:54:35.078212 27424 raft_consensus.cc:2258] T 4a7949c81c464c29929724f9543d9610 P d9c3c990e7384930a3a720c3af4458aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:54:35.078365 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 26616
I20260810 03:54:35.113616 27424 ts_tablet_manager.cc:1905] T 4a7949c81c464c29929724f9543d9610 P d9c3c990e7384930a3a720c3af4458aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:54:35.116297 27424 ts_tablet_manager.cc:1918] T 4a7949c81c464c29929724f9543d9610 P d9c3c990e7384930a3a720c3af4458aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:54:35.116403 27424 log.cc:1196] T 4a7949c81c464c29929724f9543d9610 P d9c3c990e7384930a3a720c3af4458aa: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/4a7949c81c464c29929724f9543d9610
I20260810 03:54:35.116719 27424 ts_tablet_manager.cc:1939] T 4a7949c81c464c29929724f9543d9610 P d9c3c990e7384930a3a720c3af4458aa: Deleting consensus metadata
I20260810 03:54:35.118180 26482 catalog_manager.cc:4841] TS d9c3c990e7384930a3a720c3af4458aa (127.27.153.67:39555): tablet 4a7949c81c464c29929724f9543d9610 (table test-table [id=dc6b11d66b214a739ad9afab4619f8e9]) successfully deleted
I20260810 03:54:35.119369 27428 tablet_replica.cc:330] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518: stopping tablet replica
I20260810 03:54:35.119751 27428 raft_consensus.cc:2229] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:54:35.120170 27428 raft_consensus.cc:2258] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:54:35.121490 27428 ts_tablet_manager.cc:1905] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:54:35.123805 27428 ts_tablet_manager.cc:1918] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:54:35.123908 27428 log.cc:1196] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/4a7949c81c464c29929724f9543d9610
I20260810 03:54:35.124181 27428 ts_tablet_manager.cc:1939] T 4a7949c81c464c29929724f9543d9610 P d9067b5d6a3543f0b9a6e1aaffe52518: Deleting consensus metadata
I20260810 03:54:35.125859 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 26866
I20260810 03:54:35.125933 26483 catalog_manager.cc:4841] TS d9067b5d6a3543f0b9a6e1aaffe52518 (127.27.153.66:37569): tablet 4a7949c81c464c29929724f9543d9610 (table test-table [id=dc6b11d66b214a739ad9afab4619f8e9]) successfully deleted
W20260810 03:54:35.130777 26483 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.153.65:46055 (error 108)
I20260810 03:54:35.134109 27370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:35.133812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:46055 (local address 127.0.0.1:43428)
0810 03:54:35.133936 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:35.134004 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:46055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 03:54:35.134415 26483 catalog_manager.cc:4584] TS b89b32e3545645efb377d9516935fd86 (127.27.153.65:46055): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4a7949c81c464c29929724f9543d9610: Network error: Client connection negotiation failed: client connection to 127.27.153.65:46055: connect: Connection refused (error 111)
I20260810 03:54:35.138473 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 27113
I20260810 03:54:35.156361 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 26448
2026-08-10T03:54:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2535 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T03:54:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:54:35Z Disabled control of system clock
I20260810 03:54:35.206164 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:37833
--webserver_interface=127.27.153.126
--webserver_port=0
--builtin_ntp_servers=127.27.153.84:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:37833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:54:35.323860 27438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:35.324215 27438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:35.324256 27438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:35.328424 27438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:54:35.328503 27438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:35.328517 27438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:35.328533 27438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:54:35.328548 27438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:54:35.333652 27438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:37833
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:37833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:35.334880 27438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:35.336396 27438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:35.345405 27447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:35.345443 27450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:35.345479 27438 server_base.cc:1034] running on GCE node
W20260810 03:54:35.345443 27446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:35.346402 27438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:35.347685 27438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:35.348899 27438 hybrid_clock.cc:648] HybridClock initialized: now 1786334075348868 us; error 44 us; skew 500 ppm
I20260810 03:54:35.351176 27438 webserver.cc:460] Webserver started at http://127.27.153.126:45263/ using document root <none> and password file <none>
I20260810 03:54:35.351876 27438 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:35.351959 27438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:35.352198 27438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:35.354514 27438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "66e5fd8191cb44c7ae1906c4c2f2bce8"
format_stamp: "Formatted at 2026-08-10 03:54:35 on dist-test-slave-pww0"
server_key: "e4ff27df81e6716ff465ddd492ddb172"
server_key_iv: "1ec5e5039fc25ad007fd71f903603980"
server_key_version: "encryptionkey@0"
I20260810 03:54:35.355195 27438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "66e5fd8191cb44c7ae1906c4c2f2bce8"
format_stamp: "Formatted at 2026-08-10 03:54:35 on dist-test-slave-pww0"
server_key: "e4ff27df81e6716ff465ddd492ddb172"
server_key_iv: "1ec5e5039fc25ad007fd71f903603980"
server_key_version: "encryptionkey@0"
I20260810 03:54:35.361699 27438 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260810 03:54:35.364634 27461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:35.366041 27438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:54:35.366197 27438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "66e5fd8191cb44c7ae1906c4c2f2bce8"
format_stamp: "Formatted at 2026-08-10 03:54:35 on dist-test-slave-pww0"
server_key: "e4ff27df81e6716ff465ddd492ddb172"
server_key_iv: "1ec5e5039fc25ad007fd71f903603980"
server_key_version: "encryptionkey@0"
I20260810 03:54:35.366323 27438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:35.392416 27438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:35.393208 27438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:35.393411 27438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:35.400697 27438 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:37833
I20260810 03:54:35.400748 27558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:37833 every 8 connection(s)
I20260810 03:54:35.401702 27438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:54:35.403227 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 27438
I20260810 03:54:35.403360 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:54:35.403728 28261 external_mini_cluster.cc:1428] Setting key ced50df5abcc5b45de4ff7feb8f79b58
I20260810 03:54:35.405141 27561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:35.412678 27561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8: Bootstrap starting.
I20260810 03:54:35.414924 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:35.405307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34456 (local address 127.27.153.126:37833)
0810 03:54:35.406034 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:35.406057 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:35.406102 (+ 45us) server_negotiation.cc:407] Connection header received
0810 03:54:35.406971 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:35.406997 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:35.407552 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:35.407969 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:35.408443 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:35.409924 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:35.410924 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:35.411718 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:35.411895 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:35.412728 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:35.412735 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:35.412752 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:35.413285 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:35.413364 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:35.413392 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:35.413489 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:35.413605 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 03:54:35.413909 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":130,"threads_started":1}
I20260810 03:54:35.415596 27561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:35.416374 27561 log.cc:824] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:35.418439 27561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8: No bootstrap required, opened a new log
I20260810 03:54:35.422240 27561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66e5fd8191cb44c7ae1906c4c2f2bce8" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37833 } }
I20260810 03:54:35.422530 27561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:35.422595 27561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66e5fd8191cb44c7ae1906c4c2f2bce8, State: Initialized, Role: FOLLOWER
I20260810 03:54:35.423038 27561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66e5fd8191cb44c7ae1906c4c2f2bce8" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37833 } }
I20260810 03:54:35.423192 27561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:54:35.423264 27561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:54:35.423368 27561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:35.424451 27561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66e5fd8191cb44c7ae1906c4c2f2bce8" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37833 } }
I20260810 03:54:35.424816 27561 leader_election.cc:302] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66e5fd8191cb44c7ae1906c4c2f2bce8; no voters:
I20260810 03:54:35.425150 27561 leader_election.cc:288] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:54:35.425287 27568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:54:35.425524 27568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8 [term 1 LEADER]: Becoming Leader. State: Replica: 66e5fd8191cb44c7ae1906c4c2f2bce8, State: Running, Role: LEADER
I20260810 03:54:35.425864 27568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66e5fd8191cb44c7ae1906c4c2f2bce8" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37833 } }
I20260810 03:54:35.426460 27561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:54:35.427626 27569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66e5fd8191cb44c7ae1906c4c2f2bce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66e5fd8191cb44c7ae1906c4c2f2bce8" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37833 } } }
I20260810 03:54:35.427767 27569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:35.428023 27573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66e5fd8191cb44c7ae1906c4c2f2bce8. Latest consensus state: current_term: 1 leader_uuid: "66e5fd8191cb44c7ae1906c4c2f2bce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66e5fd8191cb44c7ae1906c4c2f2bce8" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37833 } } }
I20260810 03:54:35.428122 27573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8 [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:35.428287 27587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:54:35.431648 27587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:54:35.437767 27587 catalog_manager.cc:1347] Generated new cluster ID: 4b4a241f94db4363886b9679bcc91321
I20260810 03:54:35.437858 27587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:54:35.454815 27587 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:54:35.455883 27587 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:54:35.463425 27587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8: Generated new TSK 0
I20260810 03:54:35.464450 27587 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:54:35.469602 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:37833
--builtin_ntp_servers=127.27.153.84:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:54:35.606577 27603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:35.606911 27603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:35.606969 27603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:35.611240 27603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:35.611318 27603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:35.611397 27603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:54:35.616503 27603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:37833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:35.617686 27603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:35.619134 27603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:54:35.629460 27603 server_base.cc:1034] running on GCE node
W20260810 03:54:35.629403 27614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:35.629405 27609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:35.629403 27608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:35.630191 27603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:35.630893 27603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:35.632066 27603 hybrid_clock.cc:648] HybridClock initialized: now 1786334075632042 us; error 30 us; skew 500 ppm
I20260810 03:54:35.634084 27603 webserver.cc:460] Webserver started at http://127.27.153.65:45173/ using document root <none> and password file <none>
I20260810 03:54:35.634689 27603 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:35.634768 27603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:35.634999 27603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:35.637279 27603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "3bd7ccaaf83e49c9868178a9a6f2b70f"
format_stamp: "Formatted at 2026-08-10 03:54:35 on dist-test-slave-pww0"
server_key: "a8e1d4abbdc4e1b6949565dec36aef3b"
server_key_iv: "428d8ab56d92851d7158cf3fbcc3270e"
server_key_version: "encryptionkey@0"
I20260810 03:54:35.637969 27603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "3bd7ccaaf83e49c9868178a9a6f2b70f"
format_stamp: "Formatted at 2026-08-10 03:54:35 on dist-test-slave-pww0"
server_key: "a8e1d4abbdc4e1b6949565dec36aef3b"
server_key_iv: "428d8ab56d92851d7158cf3fbcc3270e"
server_key_version: "encryptionkey@0"
I20260810 03:54:35.642168 27603 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260810 03:54:35.645002 27627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:35.646409 27603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:54:35.646549 27603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "3bd7ccaaf83e49c9868178a9a6f2b70f"
format_stamp: "Formatted at 2026-08-10 03:54:35 on dist-test-slave-pww0"
server_key: "a8e1d4abbdc4e1b6949565dec36aef3b"
server_key_iv: "428d8ab56d92851d7158cf3fbcc3270e"
server_key_version: "encryptionkey@0"
I20260810 03:54:35.646684 27603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:35.671378 27603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:35.672194 27603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:35.672386 27603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:35.673034 27603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:54:35.674073 27603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:54:35.674122 27603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:35.674181 27603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:54:35.674216 27603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:35.683333 27603 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:37931
I20260810 03:54:35.683359 27855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:37931 every 8 connection(s)
I20260810 03:54:35.684319 27603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:54:35.689939 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 27603
I20260810 03:54:35.690093 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:54:35.690446 28261 external_mini_cluster.cc:1428] Setting key 82cbfe8197eecb9cbebf4ff4e940c511
I20260810 03:54:35.692595 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:37833
--builtin_ntp_servers=127.27.153.84:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:54:35.694617 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:35.686275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:39271 (local address 127.27.153.126:37833)
0810 03:54:35.686461 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:35.686465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:35.687582 (+ 1117us) server_negotiation.cc:407] Connection header received
0810 03:54:35.689048 (+ 1466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:35.689052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:35.689119 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:35.689197 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:35.690391 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:35.691070 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:35.692568 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:35.692870 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:35.692965 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:35.694120 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:35.694124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:35.694127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:35.694143 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:35.694164 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:35.694172 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:35.694232 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:35.694382 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 03:54:35.694458 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 03:54:35.695622 27859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:35.686698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:37833 (local address 127.27.153.65:39271)
0810 03:54:35.687397 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:35.687447 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:35.688821 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:35.689394 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:35.689401 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:35.689493 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:54:35.690178 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:35.690200 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:35.691206 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:35.691210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:35.692448 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:35.692459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:35.693030 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:35.693035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:35.693210 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:35.693840 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:54:35.693842 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:54:35.693888 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:54:35.693963 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:54:35.694242 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:54:35.694253 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:54:35.694260 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:54:35.694374 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 03:54:35.694697 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":118,"threads_started":1}
I20260810 03:54:35.696883 27856 heartbeater.cc:347] Connected to a master server at 127.27.153.126:37833
I20260810 03:54:35.697161 27856 heartbeater.cc:464] Registering TS with master...
I20260810 03:54:35.697744 27856 heartbeater.cc:511] Master 127.27.153.126:37833 requested a full tablet report, sending...
I20260810 03:54:35.699329 27501 ts_manager.cc:194] Registered new tserver with Master: 3bd7ccaaf83e49c9868178a9a6f2b70f (127.27.153.65:37931)
I20260810 03:54:35.700568 27501 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.153.65:39271
W20260810 03:54:35.808395 27861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:35.808668 27861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:35.808706 27861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:35.812831 27861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:35.812896 27861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:35.812952 27861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:54:35.817814 27861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:37833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:35.818974 27861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:35.820452 27861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:35.830356 27876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:35.830364 27870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:35.830382 27872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:35.830619 27861 server_base.cc:1034] running on GCE node
I20260810 03:54:35.831048 27861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:35.831761 27861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:35.832919 27861 hybrid_clock.cc:648] HybridClock initialized: now 1786334075832891 us; error 49 us; skew 500 ppm
I20260810 03:54:35.834776 27861 webserver.cc:460] Webserver started at http://127.27.153.66:43247/ using document root <none> and password file <none>
I20260810 03:54:35.835350 27861 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:35.835422 27861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:35.835677 27861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:35.837810 27861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "3f4bd45c637f47f9b262d7db449b6508"
format_stamp: "Formatted at 2026-08-10 03:54:35 on dist-test-slave-pww0"
server_key: "5d334112054086509292746fbd4eed41"
server_key_iv: "2beb8573f51994fcd828973310dfdfed"
server_key_version: "encryptionkey@0"
I20260810 03:54:35.838384 27861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "3f4bd45c637f47f9b262d7db449b6508"
format_stamp: "Formatted at 2026-08-10 03:54:35 on dist-test-slave-pww0"
server_key: "5d334112054086509292746fbd4eed41"
server_key_iv: "2beb8573f51994fcd828973310dfdfed"
server_key_version: "encryptionkey@0"
I20260810 03:54:35.842170 27861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260810 03:54:35.844734 27889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:35.845979 27861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 03:54:35.846095 27861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "3f4bd45c637f47f9b262d7db449b6508"
format_stamp: "Formatted at 2026-08-10 03:54:35 on dist-test-slave-pww0"
server_key: "5d334112054086509292746fbd4eed41"
server_key_iv: "2beb8573f51994fcd828973310dfdfed"
server_key_version: "encryptionkey@0"
I20260810 03:54:35.846202 27861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:35.864589 27861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:35.865355 27861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:35.865543 27861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:35.866174 27861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:54:35.867182 27861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:54:35.867230 27861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:35.867285 27861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:54:35.867312 27861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:35.876276 27861 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:45409
I20260810 03:54:35.876361 28104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:45409 every 8 connection(s)
I20260810 03:54:35.877379 27861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:54:35.880420 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 27861
I20260810 03:54:35.880565 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:54:35.880877 28261 external_mini_cluster.cc:1428] Setting key 77196b382f6aac7ab8b85e459764c76b
I20260810 03:54:35.883085 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:37833
--builtin_ntp_servers=127.27.153.84:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:54:35.887799 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:35.879239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:54139 (local address 127.27.153.126:37833)
0810 03:54:35.879407 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:35.879410 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:35.880711 (+ 1301us) server_negotiation.cc:407] Connection header received
0810 03:54:35.882467 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:35.882470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:35.882523 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:35.882598 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:35.883665 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:35.884199 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:35.885756 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:35.885997 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:35.886094 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:35.887272 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:35.887275 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:35.887277 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:35.887292 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:35.887313 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:35.887321 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:35.887381 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:35.887568 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 03:54:35.887643 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:54:35.888731 28111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:35.879663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:37833 (local address 127.27.153.66:54139)
0810 03:54:35.880491 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:35.880553 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:35.882240 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:35.882799 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:35.882806 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:35.882897 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:54:35.883517 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:35.883537 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:35.884336 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:35.884340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:35.885590 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:35.885616 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:35.886173 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:35.886178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:35.886365 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:35.886983 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:54:35.886985 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:54:35.887029 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:54:35.887107 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:54:35.887393 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:54:35.887403 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:54:35.887409 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 03:54:35.887560 (+ 151us) client_negotiation.cc:240] Negotiation successful
0810 03:54:35.887891 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":114,"threads_started":1}
I20260810 03:54:35.889990 28106 heartbeater.cc:347] Connected to a master server at 127.27.153.126:37833
I20260810 03:54:35.890245 28106 heartbeater.cc:464] Registering TS with master...
I20260810 03:54:35.890841 28106 heartbeater.cc:511] Master 127.27.153.126:37833 requested a full tablet report, sending...
I20260810 03:54:35.892019 27501 ts_manager.cc:194] Registered new tserver with Master: 3f4bd45c637f47f9b262d7db449b6508 (127.27.153.66:45409)
I20260810 03:54:35.892611 27501 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.153.66:54139
W20260810 03:54:36.002764 28112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:36.003077 28112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:36.003135 28112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:36.007333 28112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:36.007422 28112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:36.007522 28112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:54:36.012573 28112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:37833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:36.013763 28112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:36.015235 28112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:36.025362 28120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:36.025373 28126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:36.025373 28122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:36.025416 28112 server_base.cc:1034] running on GCE node
I20260810 03:54:36.026062 28112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:36.026819 28112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:36.027994 28112 hybrid_clock.cc:648] HybridClock initialized: now 1786334076027953 us; error 54 us; skew 500 ppm
I20260810 03:54:36.029958 28112 webserver.cc:460] Webserver started at http://127.27.153.67:41863/ using document root <none> and password file <none>
I20260810 03:54:36.030589 28112 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:36.030666 28112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:36.030876 28112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:36.033174 28112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "ae67f53a25164a6dbc9941a93d3550f1"
format_stamp: "Formatted at 2026-08-10 03:54:36 on dist-test-slave-pww0"
server_key: "d3b854645f2b4ca6636d56e6bafd2257"
server_key_iv: "acf1e9eaba441be67e4318bf825beabe"
server_key_version: "encryptionkey@0"
I20260810 03:54:36.033792 28112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "ae67f53a25164a6dbc9941a93d3550f1"
format_stamp: "Formatted at 2026-08-10 03:54:36 on dist-test-slave-pww0"
server_key: "d3b854645f2b4ca6636d56e6bafd2257"
server_key_iv: "acf1e9eaba441be67e4318bf825beabe"
server_key_version: "encryptionkey@0"
I20260810 03:54:36.038013 28112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:54:36.040894 28144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:36.042380 28112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:54:36.042536 28112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "ae67f53a25164a6dbc9941a93d3550f1"
format_stamp: "Formatted at 2026-08-10 03:54:36 on dist-test-slave-pww0"
server_key: "d3b854645f2b4ca6636d56e6bafd2257"
server_key_iv: "acf1e9eaba441be67e4318bf825beabe"
server_key_version: "encryptionkey@0"
I20260810 03:54:36.042667 28112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:36.071153 28112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:36.072041 28112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:36.072264 28112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:36.072984 28112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:54:36.074113 28112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:54:36.074172 28112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:36.074249 28112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:54:36.074278 28112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:36.084203 28112 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:41167
I20260810 03:54:36.084229 28382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:41167 every 8 connection(s)
I20260810 03:54:36.085276 28112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:54:36.091750 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 28112
I20260810 03:54:36.091881 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:54:36.092191 28261 external_mini_cluster.cc:1428] Setting key f9927e4e7501668c49477ccc90d7087d
I20260810 03:54:36.095372 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:36.087407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:47075 (local address 127.27.153.126:37833)
0810 03:54:36.087577 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:36.087581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:36.088745 (+ 1164us) server_negotiation.cc:407] Connection header received
0810 03:54:36.090149 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:36.090152 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:36.090202 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:36.090285 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:36.091252 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.091874 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.093347 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.093630 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.093727 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:36.094897 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:36.094901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:36.094904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:36.094918 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:36.094938 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:36.094946 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:36.095004 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:36.095148 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 03:54:36.095215 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 03:54:36.096319 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:36.087853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:37833 (local address 127.27.153.67:47075)
0810 03:54:36.088578 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:36.088623 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:36.089950 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:36.090467 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:36.090474 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:36.090564 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:54:36.091111 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:36.091124 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.092043 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.092047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:36.093218 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:36.093232 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.093808 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.093814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:36.093982 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:36.094619 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:54:36.094621 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:54:36.094666 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:54:36.094736 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:54:36.095014 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:54:36.095025 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:54:36.095031 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 03:54:36.095142 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 03:54:36.095478 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":141,"threads_started":1}
I20260810 03:54:36.097549 28383 heartbeater.cc:347] Connected to a master server at 127.27.153.126:37833
I20260810 03:54:36.097841 28383 heartbeater.cc:464] Registering TS with master...
I20260810 03:54:36.098402 28383 heartbeater.cc:511] Master 127.27.153.126:37833 requested a full tablet report, sending...
I20260810 03:54:36.099501 27501 ts_manager.cc:194] Registered new tserver with Master: ae67f53a25164a6dbc9941a93d3550f1 (127.27.153.67:41167)
I20260810 03:54:36.100093 27501 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.153.67:47075
I20260810 03:54:36.106498 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:54:36.112457 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:36.108652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.27.153.126:37833)
0810 03:54:36.108785 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:36.108790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:36.108997 (+ 207us) server_negotiation.cc:407] Connection header received
0810 03:54:36.109124 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:36.109128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:36.109175 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:36.109242 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:36.109532 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.110058 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.110920 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.111139 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.111230 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:36.112022 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:36.112025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:36.112027 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:36.112039 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:36.112059 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:36.112067 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:36.112127 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:36.112243 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 03:54:36.112304 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 03:54:36.115326 27501 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34462:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:54:36.118517 27501 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:54:36.133561 28413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:36.128662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36818 (local address 127.27.153.66:45409)
0810 03:54:36.128983 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:36.128994 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:36.129016 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:54:36.129089 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:36.129094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:36.129262 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:36.129659 (+ 397us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:36.129809 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.130901 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.132188 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.132863 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.132963 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:36.133171 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:36.133274 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 03:54:36.133374 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":104,"threads_started":1}
I20260810 03:54:36.133570 28406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:36.127299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:45409 (local address 127.0.0.1:36818)
0810 03:54:36.127650 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:36.127700 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:36.127869 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:36.129416 (+ 1547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:36.129419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:36.129436 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:36.129525 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:36.129531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.131022 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.131025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:36.132062 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:36.132072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.132976 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.132979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:36.133085 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:36.133121 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 03:54:36.133237 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 03:54:36.133373 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":172,"thread_start_us":90,"threads_started":1}
I20260810 03:54:36.133581 28408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:36.128068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:37931 (local address 127.0.0.1:58610)
0810 03:54:36.128382 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:36.128395 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:36.128478 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:36.129140 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:36.129147 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:36.129172 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:36.129347 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:36.129355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.130957 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.130960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:36.132160 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:36.132168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.133158 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.133162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:36.133247 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:36.133272 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 03:54:36.133321 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 03:54:36.133398 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":60,"thread_start_us":109,"threads_started":1}
I20260810 03:54:36.133997 28407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:36.127425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:41167 (local address 127.0.0.1:58730)
0810 03:54:36.127895 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:36.127909 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:36.128001 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:36.129270 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:36.129273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:36.129291 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:36.129392 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:36.129398 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.130529 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.130532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:36.131974 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:36.131986 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.133298 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.133301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:36.133385 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:36.133404 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 03:54:36.133469 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 03:54:36.133528 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":116,"thread_start_us":114,"threads_started":1}
I20260810 03:54:36.134164 28412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:36.128286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58610 (local address 127.27.153.65:37931)
0810 03:54:36.128683 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:36.128692 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:36.128710 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:54:36.128773 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:36.128776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:36.128964 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:36.129500 (+ 536us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:36.129650 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.130832 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.132283 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.133031 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.133651 (+ 620us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:36.133855 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:36.133908 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 03:54:36.134006 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":100,"threads_started":1}
I20260810 03:54:36.134262 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:36.128208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.27.153.67:41167)
0810 03:54:36.128577 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:36.128590 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:36.128615 (+ 25us) server_negotiation.cc:407] Connection header received
0810 03:54:36.128693 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:36.128699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:36.128937 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:36.129099 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:36.129528 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.130410 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.132117 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.132928 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.133036 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:36.133229 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:36.133998 (+ 769us) server_negotiation.cc:299] Negotiation successful
0810 03:54:36.134107 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":167,"threads_started":1}
I20260810 03:54:36.136898 28254 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2689ae228a4bc58173cbbe1511f50c (DEFAULT_TABLE table=test-table [id=b746c9214d184d9f93850748aad5cc15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:54:36.136986 27721 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2689ae228a4bc58173cbbe1511f50c (DEFAULT_TABLE table=test-table [id=b746c9214d184d9f93850748aad5cc15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:54:36.136986 27975 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2689ae228a4bc58173cbbe1511f50c (DEFAULT_TABLE table=test-table [id=b746c9214d184d9f93850748aad5cc15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:54:36.137953 28254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d2689ae228a4bc58173cbbe1511f50c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:36.138058 27721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d2689ae228a4bc58173cbbe1511f50c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:36.138058 27975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d2689ae228a4bc58173cbbe1511f50c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:36.145485 28414 tablet_bootstrap.cc:492] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1: Bootstrap starting.
I20260810 03:54:36.146498 28416 tablet_bootstrap.cc:492] T 1d2689ae228a4bc58173cbbe1511f50c P 3bd7ccaaf83e49c9868178a9a6f2b70f: Bootstrap starting.
I20260810 03:54:36.147033 28417 tablet_bootstrap.cc:492] T 1d2689ae228a4bc58173cbbe1511f50c P 3f4bd45c637f47f9b262d7db449b6508: Bootstrap starting.
I20260810 03:54:36.147943 28414 tablet_bootstrap.cc:654] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:36.148715 28414 log.cc:824] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:36.149410 28416 tablet_bootstrap.cc:654] T 1d2689ae228a4bc58173cbbe1511f50c P 3bd7ccaaf83e49c9868178a9a6f2b70f: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:36.149410 28417 tablet_bootstrap.cc:654] T 1d2689ae228a4bc58173cbbe1511f50c P 3f4bd45c637f47f9b262d7db449b6508: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:36.150362 28416 log.cc:824] T 1d2689ae228a4bc58173cbbe1511f50c P 3bd7ccaaf83e49c9868178a9a6f2b70f: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:36.150362 28417 log.cc:824] T 1d2689ae228a4bc58173cbbe1511f50c P 3f4bd45c637f47f9b262d7db449b6508: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:36.151752 28414 tablet_bootstrap.cc:492] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1: No bootstrap required, opened a new log
I20260810 03:54:36.151983 28414 ts_tablet_manager.cc:1397] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 03:54:36.152773 28416 tablet_bootstrap.cc:492] T 1d2689ae228a4bc58173cbbe1511f50c P 3bd7ccaaf83e49c9868178a9a6f2b70f: No bootstrap required, opened a new log
I20260810 03:54:36.153007 28416 ts_tablet_manager.cc:1397] T 1d2689ae228a4bc58173cbbe1511f50c P 3bd7ccaaf83e49c9868178a9a6f2b70f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 03:54:36.153265 28417 tablet_bootstrap.cc:492] T 1d2689ae228a4bc58173cbbe1511f50c P 3f4bd45c637f47f9b262d7db449b6508: No bootstrap required, opened a new log
I20260810 03:54:36.153481 28417 ts_tablet_manager.cc:1397] T 1d2689ae228a4bc58173cbbe1511f50c P 3f4bd45c637f47f9b262d7db449b6508: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 03:54:36.155606 28414 raft_consensus.cc:348] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4bd45c637f47f9b262d7db449b6508" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 45409 } } peers { permanent_uuid: "ae67f53a25164a6dbc9941a93d3550f1" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41167 } } peers { permanent_uuid: "3bd7ccaaf83e49c9868178a9a6f2b70f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 37931 } }
I20260810 03:54:36.155946 28414 raft_consensus.cc:374] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:36.156008 28414 raft_consensus.cc:729] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae67f53a25164a6dbc9941a93d3550f1, State: Initialized, Role: FOLLOWER
I20260810 03:54:36.156481 28414 consensus_queue.cc:260] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4bd45c637f47f9b262d7db449b6508" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 45409 } } peers { permanent_uuid: "ae67f53a25164a6dbc9941a93d3550f1" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41167 } } peers { permanent_uuid: "3bd7ccaaf83e49c9868178a9a6f2b70f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 37931 } }
I20260810 03:54:36.157222 28383 heartbeater.cc:503] Master 127.27.153.126:37833 was elected leader, sending a full tablet report...
I20260810 03:54:36.157369 28417 raft_consensus.cc:348] T 1d2689ae228a4bc58173cbbe1511f50c P 3f4bd45c637f47f9b262d7db449b6508 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4bd45c637f47f9b262d7db449b6508" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 45409 } } peers { permanent_uuid: "ae67f53a25164a6dbc9941a93d3550f1" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41167 } } peers { permanent_uuid: "3bd7ccaaf83e49c9868178a9a6f2b70f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 37931 } }
I20260810 03:54:36.157616 28414 ts_tablet_manager.cc:1428] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:54:36.157655 28417 raft_consensus.cc:374] T 1d2689ae228a4bc58173cbbe1511f50c P 3f4bd45c637f47f9b262d7db449b6508 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:36.157719 28417 raft_consensus.cc:729] T 1d2689ae228a4bc58173cbbe1511f50c P 3f4bd45c637f47f9b262d7db449b6508 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f4bd45c637f47f9b262d7db449b6508, State: Initialized, Role: FOLLOWER
I20260810 03:54:36.158200 28417 consensus_queue.cc:260] T 1d2689ae228a4bc58173cbbe1511f50c P 3f4bd45c637f47f9b262d7db449b6508 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4bd45c637f47f9b262d7db449b6508" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 45409 } } peers { permanent_uuid: "ae67f53a25164a6dbc9941a93d3550f1" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41167 } } peers { permanent_uuid: "3bd7ccaaf83e49c9868178a9a6f2b70f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 37931 } }
I20260810 03:54:36.158073 28416 raft_consensus.cc:348] T 1d2689ae228a4bc58173cbbe1511f50c P 3bd7ccaaf83e49c9868178a9a6f2b70f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4bd45c637f47f9b262d7db449b6508" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 45409 } } peers { permanent_uuid: "ae67f53a25164a6dbc9941a93d3550f1" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41167 } } peers { permanent_uuid: "3bd7ccaaf83e49c9868178a9a6f2b70f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 37931 } }
I20260810 03:54:36.158350 28416 raft_consensus.cc:374] T 1d2689ae228a4bc58173cbbe1511f50c P 3bd7ccaaf83e49c9868178a9a6f2b70f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:36.158409 28416 raft_consensus.cc:729] T 1d2689ae228a4bc58173cbbe1511f50c P 3bd7ccaaf83e49c9868178a9a6f2b70f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd7ccaaf83e49c9868178a9a6f2b70f, State: Initialized, Role: FOLLOWER
I20260810 03:54:36.158882 28416 consensus_queue.cc:260] T 1d2689ae228a4bc58173cbbe1511f50c P 3bd7ccaaf83e49c9868178a9a6f2b70f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4bd45c637f47f9b262d7db449b6508" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 45409 } } peers { permanent_uuid: "ae67f53a25164a6dbc9941a93d3550f1" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41167 } } peers { permanent_uuid: "3bd7ccaaf83e49c9868178a9a6f2b70f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 37931 } }
I20260810 03:54:36.159056 28106 heartbeater.cc:503] Master 127.27.153.126:37833 was elected leader, sending a full tablet report...
I20260810 03:54:36.159328 28417 ts_tablet_manager.cc:1428] T 1d2689ae228a4bc58173cbbe1511f50c P 3f4bd45c637f47f9b262d7db449b6508: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 03:54:36.159931 28416 ts_tablet_manager.cc:1428] T 1d2689ae228a4bc58173cbbe1511f50c P 3bd7ccaaf83e49c9868178a9a6f2b70f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 03:54:36.159950 27856 heartbeater.cc:503] Master 127.27.153.126:37833 was elected leader, sending a full tablet report...
W20260810 03:54:36.185602 27857 tablet.cc:2369] T 1d2689ae228a4bc58173cbbe1511f50c P 3bd7ccaaf83e49c9868178a9a6f2b70f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:54:36.210491 28422 raft_consensus.cc:482] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:54:36.210722 28422 raft_consensus.cc:504] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4bd45c637f47f9b262d7db449b6508" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 45409 } } peers { permanent_uuid: "ae67f53a25164a6dbc9941a93d3550f1" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41167 } } peers { permanent_uuid: "3bd7ccaaf83e49c9868178a9a6f2b70f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 37931 } }
I20260810 03:54:36.211963 28422 leader_election.cc:288] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f4bd45c637f47f9b262d7db449b6508 (127.27.153.66:45409), 3bd7ccaaf83e49c9868178a9a6f2b70f (127.27.153.65:37931)
I20260810 03:54:36.215960 28413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:36.212441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:54385 (local address 127.27.153.66:45409)
0810 03:54:36.212593 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:36.212598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:36.212617 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:54:36.212687 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:36.212691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:36.212748 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:36.212831 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:36.213185 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.213738 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.214920 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.215372 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.215523 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:36.215580 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:36.215760 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 03:54:36.215839 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:54:36.215960 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:36.212259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:45409 (local address 127.27.153.67:54385)
0810 03:54:36.212451 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:36.212468 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:36.212588 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:36.212899 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:36.212902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:36.212923 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:36.213024 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:36.213030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.213855 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.213859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:36.214757 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:36.214768 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.215522 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.215526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:36.215639 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:36.215659 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 03:54:36.215726 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 03:54:36.215810 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 03:54:36.216516 28425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:36.212552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:37931 (local address 127.27.153.67:49397)
0810 03:54:36.212931 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:36.212941 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:36.213032 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:36.213343 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:36.213353 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:36.213375 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:36.213500 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:36.213509 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.214337 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.214340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:36.215271 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:36.215282 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.216067 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.216071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:36.216175 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:36.216198 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 03:54:36.216304 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 03:54:36.216375 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":136,"threads_started":1}
I20260810 03:54:36.216583 28412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:36.212743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:49397 (local address 127.27.153.65:37931)
0810 03:54:36.212872 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:36.212877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:36.213011 (+ 134us) server_negotiation.cc:407] Connection header received
0810 03:54:36.213130 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:36.213133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:36.213180 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:36.213245 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:36.213701 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.214207 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.215437 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.215914 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.216037 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:36.216090 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:36.216355 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 03:54:36.216454 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:54:36.216815 28018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2689ae228a4bc58173cbbe1511f50c" candidate_uuid: "ae67f53a25164a6dbc9941a93d3550f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f4bd45c637f47f9b262d7db449b6508" is_pre_election: true
I20260810 03:54:36.217142 28018 raft_consensus.cc:2454] T 1d2689ae228a4bc58173cbbe1511f50c P 3f4bd45c637f47f9b262d7db449b6508 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae67f53a25164a6dbc9941a93d3550f1 in term 0.
I20260810 03:54:36.217473 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2689ae228a4bc58173cbbe1511f50c" candidate_uuid: "ae67f53a25164a6dbc9941a93d3550f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bd7ccaaf83e49c9868178a9a6f2b70f" is_pre_election: true
I20260810 03:54:36.217752 27761 raft_consensus.cc:2454] T 1d2689ae228a4bc58173cbbe1511f50c P 3bd7ccaaf83e49c9868178a9a6f2b70f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae67f53a25164a6dbc9941a93d3550f1 in term 0.
I20260810 03:54:36.217754 28150 leader_election.cc:302] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f4bd45c637f47f9b262d7db449b6508, ae67f53a25164a6dbc9941a93d3550f1; no voters:
I20260810 03:54:36.218044 28422 raft_consensus.cc:2781] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:54:36.218154 28422 raft_consensus.cc:482] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:54:36.218202 28422 raft_consensus.cc:3037] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:36.219262 28422 raft_consensus.cc:504] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4bd45c637f47f9b262d7db449b6508" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 45409 } } peers { permanent_uuid: "ae67f53a25164a6dbc9941a93d3550f1" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41167 } } peers { permanent_uuid: "3bd7ccaaf83e49c9868178a9a6f2b70f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 37931 } }
I20260810 03:54:36.219663 28422 leader_election.cc:288] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1 [CANDIDATE]: Term 1 election: Requested vote from peers 3f4bd45c637f47f9b262d7db449b6508 (127.27.153.66:45409), 3bd7ccaaf83e49c9868178a9a6f2b70f (127.27.153.65:37931)
I20260810 03:54:36.220126 28018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2689ae228a4bc58173cbbe1511f50c" candidate_uuid: "ae67f53a25164a6dbc9941a93d3550f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f4bd45c637f47f9b262d7db449b6508"
I20260810 03:54:36.220207 27761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2689ae228a4bc58173cbbe1511f50c" candidate_uuid: "ae67f53a25164a6dbc9941a93d3550f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bd7ccaaf83e49c9868178a9a6f2b70f"
I20260810 03:54:36.220252 28018 raft_consensus.cc:3037] T 1d2689ae228a4bc58173cbbe1511f50c P 3f4bd45c637f47f9b262d7db449b6508 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:36.220362 27761 raft_consensus.cc:3037] T 1d2689ae228a4bc58173cbbe1511f50c P 3bd7ccaaf83e49c9868178a9a6f2b70f [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:36.221376 28018 raft_consensus.cc:2454] T 1d2689ae228a4bc58173cbbe1511f50c P 3f4bd45c637f47f9b262d7db449b6508 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae67f53a25164a6dbc9941a93d3550f1 in term 1.
I20260810 03:54:36.221590 27761 raft_consensus.cc:2454] T 1d2689ae228a4bc58173cbbe1511f50c P 3bd7ccaaf83e49c9868178a9a6f2b70f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae67f53a25164a6dbc9941a93d3550f1 in term 1.
I20260810 03:54:36.221720 28150 leader_election.cc:302] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f4bd45c637f47f9b262d7db449b6508, ae67f53a25164a6dbc9941a93d3550f1; no voters:
I20260810 03:54:36.221912 28422 raft_consensus.cc:2781] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:54:36.222160 28422 raft_consensus.cc:686] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1 [term 1 LEADER]: Becoming Leader. State: Replica: ae67f53a25164a6dbc9941a93d3550f1, State: Running, Role: LEADER
I20260810 03:54:36.222525 28422 consensus_queue.cc:237] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4bd45c637f47f9b262d7db449b6508" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 45409 } } peers { permanent_uuid: "ae67f53a25164a6dbc9941a93d3550f1" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41167 } } peers { permanent_uuid: "3bd7ccaaf83e49c9868178a9a6f2b70f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 37931 } }
I20260810 03:54:36.225854 27500 catalog_manager.cc:5495] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae67f53a25164a6dbc9941a93d3550f1 (127.27.153.67). New cstate: current_term: 1 leader_uuid: "ae67f53a25164a6dbc9941a93d3550f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f4bd45c637f47f9b262d7db449b6508" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 45409 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae67f53a25164a6dbc9941a93d3550f1" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3bd7ccaaf83e49c9868178a9a6f2b70f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 37931 } health_report { overall_health: UNKNOWN } } }
I20260810 03:54:36.259570 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:36.255797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58742 (local address 127.27.153.67:41167)
0810 03:54:36.255918 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:36.255921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:36.255936 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:54:36.256071 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:36.256074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:36.256133 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:36.256234 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:54:36.256521 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.257056 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.257961 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:36.258220 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:36.258315 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:36.258621 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:54:36.259127 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:54:36.259242 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:36.259364 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 03:54:36.259425 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 03:54:36.266275 27761 raft_consensus.cc:1261] T 1d2689ae228a4bc58173cbbe1511f50c P 3bd7ccaaf83e49c9868178a9a6f2b70f [term 1 FOLLOWER]: Refusing update from remote peer ae67f53a25164a6dbc9941a93d3550f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:54:36.266275 28018 raft_consensus.cc:1261] T 1d2689ae228a4bc58173cbbe1511f50c P 3f4bd45c637f47f9b262d7db449b6508 [term 1 FOLLOWER]: Refusing update from remote peer ae67f53a25164a6dbc9941a93d3550f1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:54:36.267024 28422 consensus_queue.cc:1035] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bd7ccaaf83e49c9868178a9a6f2b70f" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 37931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:54:36.267271 28429 consensus_queue.cc:1035] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f4bd45c637f47f9b262d7db449b6508" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 45409 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:54:36.277030 28435 mvcc.cc:205] Tried to move back new op lower bound from 7316824376378421248 to 7316824376210341888. Current Snapshot: MvccSnapshot[applied={T|T < 7316824376378421248}]
I20260810 03:54:36.284021 27500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:54:36.284206 27500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34462:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:54:36.286985 27500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66e5fd8191cb44c7ae1906c4c2f2bce8: Sending DeleteTablet for 3 replicas of tablet 1d2689ae228a4bc58173cbbe1511f50c
I20260810 03:54:36.287976 28254 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d2689ae228a4bc58173cbbe1511f50c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:54:36 UTC) from {username='slave'} at 127.0.0.1:58730
I20260810 03:54:36.287979 27721 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d2689ae228a4bc58173cbbe1511f50c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:54:36 UTC) from {username='slave'} at 127.0.0.1:58610
I20260810 03:54:36.288033 27975 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d2689ae228a4bc58173cbbe1511f50c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:54:36 UTC) from {username='slave'} at 127.0.0.1:36818
I20260810 03:54:36.288559 28454 tablet_replica.cc:330] T 1d2689ae228a4bc58173cbbe1511f50c P 3f4bd45c637f47f9b262d7db449b6508: stopping tablet replica
I20260810 03:54:36.288559 28453 tablet_replica.cc:330] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1: stopping tablet replica
I20260810 03:54:36.288918 28454 raft_consensus.cc:2229] T 1d2689ae228a4bc58173cbbe1511f50c P 3f4bd45c637f47f9b262d7db449b6508 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:54:36.288918 28453 raft_consensus.cc:2229] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:54:36.289271 28453 raft_consensus.cc:2258] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:54:36.289268 28454 raft_consensus.cc:2258] T 1d2689ae228a4bc58173cbbe1511f50c P 3f4bd45c637f47f9b262d7db449b6508 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:54:36.289642 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 27603
I20260810 03:54:36.289656 28452 tablet_replica.cc:330] T 1d2689ae228a4bc58173cbbe1511f50c P 3bd7ccaaf83e49c9868178a9a6f2b70f: stopping tablet replica
I20260810 03:54:36.289980 28452 raft_consensus.cc:2229] T 1d2689ae228a4bc58173cbbe1511f50c P 3bd7ccaaf83e49c9868178a9a6f2b70f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:54:36.291013 28454 ts_tablet_manager.cc:1905] T 1d2689ae228a4bc58173cbbe1511f50c P 3f4bd45c637f47f9b262d7db449b6508: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:54:36.293648 28454 ts_tablet_manager.cc:1918] T 1d2689ae228a4bc58173cbbe1511f50c P 3f4bd45c637f47f9b262d7db449b6508: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:54:36.293744 28454 log.cc:1196] T 1d2689ae228a4bc58173cbbe1511f50c P 3f4bd45c637f47f9b262d7db449b6508: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/1d2689ae228a4bc58173cbbe1511f50c
I20260810 03:54:36.294044 28454 ts_tablet_manager.cc:1939] T 1d2689ae228a4bc58173cbbe1511f50c P 3f4bd45c637f47f9b262d7db449b6508: Deleting consensus metadata
I20260810 03:54:36.294359 28453 ts_tablet_manager.cc:1905] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:54:36.295501 27469 catalog_manager.cc:4841] TS 3f4bd45c637f47f9b262d7db449b6508 (127.27.153.66:45409): tablet 1d2689ae228a4bc58173cbbe1511f50c (table test-table [id=b746c9214d184d9f93850748aad5cc15]) successfully deleted
I20260810 03:54:36.296865 28453 ts_tablet_manager.cc:1918] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:54:36.296955 28453 log.cc:1196] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/1d2689ae228a4bc58173cbbe1511f50c
I20260810 03:54:36.297194 28453 ts_tablet_manager.cc:1939] T 1d2689ae228a4bc58173cbbe1511f50c P ae67f53a25164a6dbc9941a93d3550f1: Deleting consensus metadata
I20260810 03:54:36.298255 27474 catalog_manager.cc:4841] TS ae67f53a25164a6dbc9941a93d3550f1 (127.27.153.67:41167): tablet 1d2689ae228a4bc58173cbbe1511f50c (table test-table [id=b746c9214d184d9f93850748aad5cc15]) successfully deleted
W20260810 03:54:36.302215 27469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.153.65:37931 (error 108)
I20260810 03:54:36.303154 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 27861
I20260810 03:54:36.303383 28407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:36.303047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:37931 (local address 127.0.0.1:58614)
0810 03:54:36.303215 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:36.303294 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:37931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 03:54:36.334661 27469 catalog_manager.cc:4584] TS 3bd7ccaaf83e49c9868178a9a6f2b70f (127.27.153.65:37931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d2689ae228a4bc58173cbbe1511f50c: Network error: Client connection negotiation failed: client connection to 127.27.153.65:37931: connect: Connection refused (error 111)
I20260810 03:54:36.343943 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 28112
I20260810 03:54:36.353080 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 27438
2026-08-10T03:54:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1190 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:54:36 dist-test-slave-pww0 krb5kdc[28459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:54:36 dist-test-slave-pww0 krb5kdc[28459](info): set up 2 sockets
Aug 10 03:54:36 dist-test-slave-pww0 krb5kdc[28459](info): commencing operation
krb5kdc: starting...
W20260810 03:54:37.562280 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:54:37 dist-test-slave-pww0 krb5kdc[28459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:54:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:54:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:54:37.698592 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:40493
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:39621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:40493
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:54:37.816252 28475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:37.816546 28475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:37.816591 28475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:37.820744 28475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:54:37.820827 28475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:37.820847 28475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:37.820863 28475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:54:37.820876 28475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:54:37.826507 28475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:39621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:40493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:40493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:37.827736 28475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:37.828835 28475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:37.837779 28483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:37.837816 28482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:37.837880 28475 server_base.cc:1034] running on GCE node
W20260810 03:54:37.837816 28487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:37.838675 28475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:37.839975 28475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:37.841183 28475 hybrid_clock.cc:648] HybridClock initialized: now 1786334077841150 us; error 45 us; skew 500 ppm
Aug 10 03:54:37 dist-test-slave-pww0 krb5kdc[28459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334077, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:37.847800 28475 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:54:37.848889 28475 webserver.cc:460] Webserver started at http://127.27.153.126:44935/ using document root <none> and password file <none>
I20260810 03:54:37.849506 28475 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:37.849577 28475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:37.849887 28475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:37.852319 28475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "eb1999fb9f0149c9a7b79cb78c885d0d"
format_stamp: "Formatted at 2026-08-10 03:54:37 on dist-test-slave-pww0"
server_key: "b2df55b008c138d06acb9b2d39e6bafd"
server_key_iv: "9271abb897d440f855cec6e4535201eb"
server_key_version: "encryptionkey@0"
I20260810 03:54:37.852947 28475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "eb1999fb9f0149c9a7b79cb78c885d0d"
format_stamp: "Formatted at 2026-08-10 03:54:37 on dist-test-slave-pww0"
server_key: "b2df55b008c138d06acb9b2d39e6bafd"
server_key_iv: "9271abb897d440f855cec6e4535201eb"
server_key_version: "encryptionkey@0"
I20260810 03:54:37.857219 28475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:54:37.859978 28499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:37.861454 28475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:54:37.861600 28475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "eb1999fb9f0149c9a7b79cb78c885d0d"
format_stamp: "Formatted at 2026-08-10 03:54:37 on dist-test-slave-pww0"
server_key: "b2df55b008c138d06acb9b2d39e6bafd"
server_key_iv: "9271abb897d440f855cec6e4535201eb"
server_key_version: "encryptionkey@0"
I20260810 03:54:37.861728 28475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:37.883369 28475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:37.886924 28475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:37.887128 28475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:37.894909 28475 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:40493
I20260810 03:54:37.894932 28600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:40493 every 8 connection(s)
I20260810 03:54:37.896042 28475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:54:37.896538 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 28475
I20260810 03:54:37.896699 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:54:37.896996 28261 external_mini_cluster.cc:1428] Setting key 98f57f9a22eb12fa40e1b10713cc90d7
I20260810 03:54:37.899201 28601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:37.905344 28601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d: Bootstrap starting.
Aug 10 03:54:37 dist-test-slave-pww0 krb5kdc[28459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:54:37.908128 28601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:37.908913 28601 log.cc:824] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:37.911118 28601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d: No bootstrap required, opened a new log
I20260810 03:54:37.911435 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:37.898490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33802 (local address 127.27.153.126:40493)
0810 03:54:37.898938 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:37.898955 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:37.898993 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:54:37.899736 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:37.899755 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:37.900099 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:37.900437 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:37.901300 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:37.902339 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:37.903237 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:37.903778 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:37.903932 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:37.906627 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:37.906633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:37.906647 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:37.906700 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:37.908415 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:37.908864 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:37.908869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:37.908874 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:37.909000 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:37.909334 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:37.909337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:37.909339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:37.909820 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:37.909997 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:37.910332 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 03:54:37.910644 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":302,"thread_start_us":143,"threads_started":1}
I20260810 03:54:37.915337 28601 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb1999fb9f0149c9a7b79cb78c885d0d" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 40493 } }
I20260810 03:54:37.915652 28601 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:37.915714 28601 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb1999fb9f0149c9a7b79cb78c885d0d, State: Initialized, Role: FOLLOWER
I20260810 03:54:37.916136 28601 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb1999fb9f0149c9a7b79cb78c885d0d" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 40493 } }
I20260810 03:54:37.916271 28601 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:54:37.916335 28601 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:54:37.916435 28601 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:37.917419 28601 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb1999fb9f0149c9a7b79cb78c885d0d" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 40493 } }
I20260810 03:54:37.917768 28601 leader_election.cc:302] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb1999fb9f0149c9a7b79cb78c885d0d; no voters:
I20260810 03:54:37.918085 28601 leader_election.cc:288] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:54:37.918223 28612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:54:37.918478 28612 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d [term 1 LEADER]: Becoming Leader. State: Replica: eb1999fb9f0149c9a7b79cb78c885d0d, State: Running, Role: LEADER
I20260810 03:54:37.918799 28612 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb1999fb9f0149c9a7b79cb78c885d0d" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 40493 } }
I20260810 03:54:37.919371 28601 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:54:37.920480 28618 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb1999fb9f0149c9a7b79cb78c885d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb1999fb9f0149c9a7b79cb78c885d0d" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 40493 } } }
I20260810 03:54:37.920635 28618 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:37.920580 28619 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb1999fb9f0149c9a7b79cb78c885d0d. Latest consensus state: current_term: 1 leader_uuid: "eb1999fb9f0149c9a7b79cb78c885d0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb1999fb9f0149c9a7b79cb78c885d0d" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 40493 } } }
I20260810 03:54:37.920849 28619 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:37.921089 28631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:54:37.924606 28631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:54:37.930995 28631 catalog_manager.cc:1347] Generated new cluster ID: 3fa4cb6ca6e7432984391be067cade47
I20260810 03:54:37.931124 28631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:54:37.944329 28631 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:54:37.945719 28631 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:54:37.964545 28631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d: Generated new TSK 0
I20260810 03:54:37.965538 28631 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.153.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
I20260810 03:54:37.996668 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:40493
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:39621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:54:38.116583 28650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:38.116894 28650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:38.116940 28650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:38.121205 28650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:38.121290 28650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:38.121363 28650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:54:38.127182 28650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:39621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:40493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:38.128463 28650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:38.129585 28650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:38.139802 28662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:38.139943 28665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:38.139842 28660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:38.140242 28650 server_base.cc:1034] running on GCE node
I20260810 03:54:38.140715 28650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:38.141435 28650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:38.142618 28650 hybrid_clock.cc:648] HybridClock initialized: now 1786334078142595 us; error 37 us; skew 500 ppm
Aug 10 03:54:38 dist-test-slave-pww0 krb5kdc[28459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334078, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:38.145718 28650 init.cc:377] Logged in from keytab as kudu/127.27.153.65@KRBTEST.COM (short username kudu)
I20260810 03:54:38.147010 28650 webserver.cc:460] Webserver started at http://127.27.153.65:43193/ using document root <none> and password file <none>
I20260810 03:54:38.147794 28650 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:38.147881 28650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:38.148108 28650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:38.150349 28650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "23bd8d6fb9eb4b659de598726841da47"
format_stamp: "Formatted at 2026-08-10 03:54:38 on dist-test-slave-pww0"
server_key: "61a4cd7829cc2dbcb81c903b6df8f5a9"
server_key_iv: "6ef354240c362ab92206d6046bb66b5c"
server_key_version: "encryptionkey@0"
I20260810 03:54:38.150988 28650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "23bd8d6fb9eb4b659de598726841da47"
format_stamp: "Formatted at 2026-08-10 03:54:38 on dist-test-slave-pww0"
server_key: "61a4cd7829cc2dbcb81c903b6df8f5a9"
server_key_iv: "6ef354240c362ab92206d6046bb66b5c"
server_key_version: "encryptionkey@0"
I20260810 03:54:38.155258 28650 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260810 03:54:38.158039 28674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:38.159536 28650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:54:38.159698 28650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "23bd8d6fb9eb4b659de598726841da47"
format_stamp: "Formatted at 2026-08-10 03:54:38 on dist-test-slave-pww0"
server_key: "61a4cd7829cc2dbcb81c903b6df8f5a9"
server_key_iv: "6ef354240c362ab92206d6046bb66b5c"
server_key_version: "encryptionkey@0"
I20260810 03:54:38.159821 28650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:38.200434 28650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:38.203934 28650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:38.204142 28650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:38.204840 28650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:54:38.205770 28650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:54:38.205823 28650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:38.205883 28650 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:54:38.205909 28650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:38.215175 28650 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:39923
I20260810 03:54:38.215238 28874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:39923 every 8 connection(s)
I20260810 03:54:38.216279 28650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:54:38.225481 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 28650
I20260810 03:54:38.225615 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:54:38.225939 28261 external_mini_cluster.cc:1428] Setting key 4b8ee75203e607969236ba1147d2df83
Aug 10 03:54:38 dist-test-slave-pww0 krb5kdc[28459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334078, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:54:38.231132 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:38.218235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:60181 (local address 127.27.153.126:40493)
0810 03:54:38.218373 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:38.218377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:38.219307 (+ 930us) server_negotiation.cc:407] Connection header received
0810 03:54:38.220756 (+ 1449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:38.220759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:38.220822 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:38.220901 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:38.222108 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.222655 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.223957 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.224198 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.224282 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:38.227084 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:38.227087 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:38.227089 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:38.227116 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:38.228796 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:38.229344 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:38.229347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:38.229349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:38.229450 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:38.229883 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:38.229887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:38.229889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:38.230106 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:38.230202 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:38.230801 (+ 599us) server_negotiation.cc:299] Negotiation successful
0810 03:54:38.230956 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 03:54:38.232141 28877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:38.218607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:40493 (local address 127.27.153.65:60181)
0810 03:54:38.219106 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:38.219155 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:38.220542 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:38.221063 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:38.221070 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:38.221468 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:54:38.221979 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:38.221993 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.222798 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.222802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:38.223847 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:38.223858 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.224341 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.224344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:38.224502 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:38.225121 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:54:38.225141 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:54:38.226925 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:54:38.228922 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:54:38.228928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:54:38.228945 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:54:38.229245 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:54:38.229571 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:54:38.229574 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:54:38.229575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:54:38.229761 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:54:38.230234 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:54:38.230242 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:54:38.230548 (+ 306us) client_negotiation.cc:769] Sending connection context
0810 03:54:38.230735 (+ 187us) client_negotiation.cc:240] Negotiation successful
0810 03:54:38.231115 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":120,"threads_started":1}
I20260810 03:54:38.233495 28875 heartbeater.cc:347] Connected to a master server at 127.27.153.126:40493
I20260810 03:54:38.233798 28875 heartbeater.cc:464] Registering TS with master...
I20260810 03:54:38.234488 28875 heartbeater.cc:511] Master 127.27.153.126:40493 requested a full tablet report, sending...
I20260810 03:54:38.236253 28532 ts_manager.cc:194] Registered new tserver with Master: 23bd8d6fb9eb4b659de598726841da47 (127.27.153.65:39923)
WARNING: no policy specified for kudu/127.27.153.66@KRBTEST.COM; defaulting to no policy
I20260810 03:54:38.237457 28532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:60181
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
I20260810 03:54:38.256994 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:40493
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:39621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:54:38.374576 28883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:38.374877 28883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:38.374925 28883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:38.379143 28883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:38.379232 28883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:38.379299 28883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:54:38.385035 28883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:39621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:40493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:38.386264 28883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:38.387418 28883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:38.397611 28895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:38.397589 28908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:38.397632 28883 server_base.cc:1034] running on GCE node
W20260810 03:54:38.397585 28893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:38.398274 28883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:38.398943 28883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:38.400111 28883 hybrid_clock.cc:648] HybridClock initialized: now 1786334078400095 us; error 27 us; skew 500 ppm
Aug 10 03:54:38 dist-test-slave-pww0 krb5kdc[28459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334078, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:38.402966 28883 init.cc:377] Logged in from keytab as kudu/127.27.153.66@KRBTEST.COM (short username kudu)
I20260810 03:54:38.404013 28883 webserver.cc:460] Webserver started at http://127.27.153.66:41101/ using document root <none> and password file <none>
I20260810 03:54:38.404601 28883 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:38.404673 28883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:38.404874 28883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:38.407058 28883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "2888240ecb554ea2ac709a175997d1ff"
format_stamp: "Formatted at 2026-08-10 03:54:38 on dist-test-slave-pww0"
server_key: "a4934e58039af6695662f8421ccf77af"
server_key_iv: "0536631b8142eb895a3c8e72e5f882bc"
server_key_version: "encryptionkey@0"
I20260810 03:54:38.407689 28883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "2888240ecb554ea2ac709a175997d1ff"
format_stamp: "Formatted at 2026-08-10 03:54:38 on dist-test-slave-pww0"
server_key: "a4934e58039af6695662f8421ccf77af"
server_key_iv: "0536631b8142eb895a3c8e72e5f882bc"
server_key_version: "encryptionkey@0"
I20260810 03:54:38.411955 28883 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:54:38.414685 28916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:38.416146 28883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:54:38.416285 28883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "2888240ecb554ea2ac709a175997d1ff"
format_stamp: "Formatted at 2026-08-10 03:54:38 on dist-test-slave-pww0"
server_key: "a4934e58039af6695662f8421ccf77af"
server_key_iv: "0536631b8142eb895a3c8e72e5f882bc"
server_key_version: "encryptionkey@0"
I20260810 03:54:38.416401 28883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:38.430712 28883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:38.434326 28883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:38.434546 28883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:38.435189 28883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:54:38.436108 28883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:54:38.436159 28883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:38.436215 28883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:54:38.436244 28883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:38.444973 28883 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:40619
I20260810 03:54:38.445021 29111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:40619 every 8 connection(s)
I20260810 03:54:38.445933 28883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:54:38.455179 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 28883
I20260810 03:54:38.455317 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:54:38.455649 28261 external_mini_cluster.cc:1428] Setting key 8eb9647229b0dc437c48d26836e55d85
Aug 10 03:54:38 dist-test-slave-pww0 krb5kdc[28459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334078, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:54:38.460408 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:38.447864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:53721 (local address 127.27.153.126:40493)
0810 03:54:38.447994 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:38.447997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:38.448898 (+ 901us) server_negotiation.cc:407] Connection header received
0810 03:54:38.450234 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:38.450237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:38.450309 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:38.450393 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:38.451654 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.452189 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.453414 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.453672 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.453757 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:38.456554 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:38.456558 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:38.456561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:38.456591 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:38.458047 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:38.458698 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:38.458702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:38.458703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:38.458793 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:38.459179 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:38.459182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:38.459183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:38.459405 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:38.459554 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:38.460079 (+ 525us) server_negotiation.cc:299] Negotiation successful
0810 03:54:38.460230 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 03:54:38.461241 29122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:38.448223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:40493 (local address 127.27.153.66:53721)
0810 03:54:38.448722 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:38.448772 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:38.449995 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:38.450557 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:38.450565 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:38.450935 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:54:38.451444 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:38.451498 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.452319 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.452322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:38.453289 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:38.453299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.453824 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.453827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:38.453979 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:38.454586 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:54:38.454605 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:54:38.456381 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:54:38.458177 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:54:38.458186 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:54:38.458206 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:54:38.458575 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:54:38.458884 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:54:38.458887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:54:38.458889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:54:38.459088 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:54:38.459549 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:54:38.459557 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:54:38.459840 (+ 283us) client_negotiation.cc:769] Sending connection context
0810 03:54:38.460008 (+ 168us) client_negotiation.cc:240] Negotiation successful
0810 03:54:38.460306 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":116,"threads_started":1}
I20260810 03:54:38.462821 29116 heartbeater.cc:347] Connected to a master server at 127.27.153.126:40493
I20260810 03:54:38.463161 29116 heartbeater.cc:464] Registering TS with master...
I20260810 03:54:38.463932 29116 heartbeater.cc:511] Master 127.27.153.126:40493 requested a full tablet report, sending...
I20260810 03:54:38.465373 28532 ts_manager.cc:194] Registered new tserver with Master: 2888240ecb554ea2ac709a175997d1ff (127.27.153.66:40619)
I20260810 03:54:38.466017 28532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:53721
WARNING: no policy specified for kudu/127.27.153.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
I20260810 03:54:38.486912 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:40493
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:39621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:54:38.602178 29130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:38.602473 29130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:38.602519 29130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:38.606643 29130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:38.606731 29130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:38.606792 29130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:54:38.612437 29130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:39621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:40493
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:38.613583 29130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:38.614637 29130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:38.624683 29143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:38.624683 29137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:38.624693 29141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:38.625012 29130 server_base.cc:1034] running on GCE node
I20260810 03:54:38.625468 29130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:38.626153 29130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:38.627317 29130 hybrid_clock.cc:648] HybridClock initialized: now 1786334078627293 us; error 38 us; skew 500 ppm
Aug 10 03:54:38 dist-test-slave-pww0 krb5kdc[28459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334078, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:38.630053 29130 init.cc:377] Logged in from keytab as kudu/127.27.153.67@KRBTEST.COM (short username kudu)
I20260810 03:54:38.630976 29130 webserver.cc:460] Webserver started at http://127.27.153.67:46759/ using document root <none> and password file <none>
I20260810 03:54:38.631565 29130 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:38.631639 29130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:38.631846 29130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:38.634007 29130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "4af47fa8316341f2b1f06b8cd5d43111"
format_stamp: "Formatted at 2026-08-10 03:54:38 on dist-test-slave-pww0"
server_key: "741337b17ffa28bbcbf04a73899be545"
server_key_iv: "55975941b2b8617b629b73626c509afd"
server_key_version: "encryptionkey@0"
I20260810 03:54:38.634588 29130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "4af47fa8316341f2b1f06b8cd5d43111"
format_stamp: "Formatted at 2026-08-10 03:54:38 on dist-test-slave-pww0"
server_key: "741337b17ffa28bbcbf04a73899be545"
server_key_iv: "55975941b2b8617b629b73626c509afd"
server_key_version: "encryptionkey@0"
I20260810 03:54:38.638381 29130 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:54:38.640928 29159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:38.642249 29130 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 03:54:38.642387 29130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "4af47fa8316341f2b1f06b8cd5d43111"
format_stamp: "Formatted at 2026-08-10 03:54:38 on dist-test-slave-pww0"
server_key: "741337b17ffa28bbcbf04a73899be545"
server_key_iv: "55975941b2b8617b629b73626c509afd"
server_key_version: "encryptionkey@0"
I20260810 03:54:38.642504 29130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:38.664865 29130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:38.668529 29130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:38.668754 29130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:38.669386 29130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:54:38.670254 29130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:54:38.670317 29130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:38.670382 29130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:54:38.670414 29130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:38.679194 29130 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:34365
I20260810 03:54:38.679242 29376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:34365 every 8 connection(s)
I20260810 03:54:38.680190 29130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:54:38.685240 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 29130
I20260810 03:54:38.685364 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:54:38.685623 28261 external_mini_cluster.cc:1428] Setting key 5e391d9b55d00291e1da6059a3b1cf6f
Aug 10 03:54:38 dist-test-slave-pww0 krb5kdc[28459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334078, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:54:38.694442 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:38.682127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:41421 (local address 127.27.153.126:40493)
0810 03:54:38.682278 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:38.682282 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:38.683166 (+ 884us) server_negotiation.cc:407] Connection header received
0810 03:54:38.684526 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:38.684529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:38.684613 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:38.684687 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:38.685970 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.686513 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.688163 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.688392 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.688478 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:38.691057 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:38.691060 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:38.691063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:38.691092 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:38.692471 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:38.693004 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:38.693006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:38.693008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:38.693081 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:38.693418 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:38.693420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:38.693421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:38.693597 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:38.693676 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:38.694191 (+ 515us) server_negotiation.cc:299] Negotiation successful
0810 03:54:38.694310 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 03:54:38.695312 29383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:38.682499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:40493 (local address 127.27.153.67:41421)
0810 03:54:38.683017 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:38.683061 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:38.684309 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:38.684875 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:38.684882 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:38.685257 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:54:38.685820 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:38.685836 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.686667 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.686670 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:38.688053 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:38.688064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.688514 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.688516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:38.688667 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:38.689233 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:54:38.689250 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:54:38.690874 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:54:38.692610 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:54:38.692616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:54:38.692630 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:54:38.692921 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:54:38.693186 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:54:38.693189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:54:38.693190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:54:38.693342 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:54:38.693678 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:54:38.693683 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:54:38.693945 (+ 262us) client_negotiation.cc:769] Sending connection context
0810 03:54:38.694106 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 03:54:38.694387 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":131,"threads_started":1}
I20260810 03:54:38.696643 29377 heartbeater.cc:347] Connected to a master server at 127.27.153.126:40493
I20260810 03:54:38.696897 29377 heartbeater.cc:464] Registering TS with master...
I20260810 03:54:38.697423 29377 heartbeater.cc:511] Master 127.27.153.126:40493 requested a full tablet report, sending...
I20260810 03:54:38.698531 28532 ts_manager.cc:194] Registered new tserver with Master: 4af47fa8316341f2b1f06b8cd5d43111 (127.27.153.67:34365)
I20260810 03:54:38.699218 28532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:41421
I20260810 03:54:38.699884 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:54:38.709406 28608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:38.701900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33810 (local address 127.27.153.126:40493)
0810 03:54:38.702054 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:38.702059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:38.702213 (+ 154us) server_negotiation.cc:407] Connection header received
0810 03:54:38.702314 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:38.702317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:38.702369 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:38.702426 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:38.703107 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.703712 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.704535 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.704762 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.704845 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:38.706387 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:38.706390 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:38.706393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:38.706418 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:38.707673 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:38.708211 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:38.708214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:38.708215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:38.708314 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:38.708603 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:38.708605 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:38.708606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:38.708822 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:38.708937 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:38.709134 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 03:54:38.709237 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 03:54:38.712649 28531 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:54:38.715775 28531 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:54:38.731073 29412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:38.725343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55070 (local address 127.27.153.66:40619)
0810 03:54:38.725796 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:38.725809 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:38.726308 (+ 499us) server_negotiation.cc:407] Connection header received
0810 03:54:38.726382 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:38.726386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:38.726563 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:38.726705 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:38.727162 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.728244 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.729570 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.730403 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.730519 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:38.730690 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:38.730814 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 03:54:38.730916 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":147,"threads_started":1}
I20260810 03:54:38.731091 29411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:38.725086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:40619 (local address 127.0.0.1:55070)
0810 03:54:38.726115 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:38.726127 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:38.726202 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:38.726857 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:38.726860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:38.726890 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:38.727032 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:38.727038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.728395 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.728399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:38.729447 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:38.729457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.730528 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.730531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:38.730610 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:38.730649 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 03:54:38.730783 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 03:54:38.730940 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":976,"mutex_wait_us":74,"spinlock_wait_cycles":3328,"thread_start_us":102,"threads_started":1}
I20260810 03:54:38.731173 29413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:38.725562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56334 (local address 127.27.153.67:34365)
0810 03:54:38.726099 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:38.726111 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:38.726137 (+ 26us) server_negotiation.cc:407] Connection header received
0810 03:54:38.726213 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:38.726219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:38.726386 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:38.726527 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:38.726812 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.727916 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.729746 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.730582 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.730680 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:38.730802 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:38.730956 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 03:54:38.731053 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":152,"threads_started":1}
I20260810 03:54:38.731856 29407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:38.724876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:34365 (local address 127.0.0.1:56334)
0810 03:54:38.725938 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:38.725949 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:38.726031 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:38.726541 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:38.726547 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:38.726569 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:38.726702 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:38.726709 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.728050 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.728053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:38.729621 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:38.729633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.730689 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.730692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:38.730773 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:38.730797 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:54:38.730857 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 03:54:38.731704 (+ 847us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1002,"mutex_wait_us":857,"thread_start_us":106,"threads_started":1}
I20260810 03:54:38.732899 29405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:38.724742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:39923 (local address 127.0.0.1:54166)
0810 03:54:38.725698 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:38.725752 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:38.725888 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:38.726857 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:38.726860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:38.726889 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:38.727026 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:38.727033 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.728072 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.728075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:38.729635 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:38.729643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.732548 (+ 2905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.732551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:38.732615 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:38.732634 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 03:54:38.732698 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 03:54:38.732768 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":227,"spinlock_wait_cycles":6144,"thread_start_us":105,"threads_started":1}
I20260810 03:54:38.733469 29408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:38.724891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54166 (local address 127.27.153.65:39923)
0810 03:54:38.726263 (+ 1372us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:38.726279 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:38.726301 (+ 22us) server_negotiation.cc:407] Connection header received
0810 03:54:38.726373 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:38.726378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:38.726563 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:38.726705 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:38.727170 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.727944 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.730865 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.732276 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.732380 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:38.732496 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:38.733229 (+ 733us) server_negotiation.cc:299] Negotiation successful
0810 03:54:38.733335 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1268,"thread_start_us":82,"threads_started":1}
I20260810 03:54:38.734254 28993 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf40db8f527448ebddf019de532025d (DEFAULT_TABLE table=test-table [id=18fa20cce7ba4355b9c47ae419f119c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:54:38.734527 29249 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf40db8f527448ebddf019de532025d (DEFAULT_TABLE table=test-table [id=18fa20cce7ba4355b9c47ae419f119c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:54:38.735256 28993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf40db8f527448ebddf019de532025d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:38.735436 29249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf40db8f527448ebddf019de532025d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:38.741775 29415 tablet_bootstrap.cc:492] T 2cf40db8f527448ebddf019de532025d P 4af47fa8316341f2b1f06b8cd5d43111: Bootstrap starting.
I20260810 03:54:38.742434 28764 tablet_service.cc:1467] Processing CreateTablet for tablet 2cf40db8f527448ebddf019de532025d (DEFAULT_TABLE table=test-table [id=18fa20cce7ba4355b9c47ae419f119c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:54:38.743249 29417 tablet_bootstrap.cc:492] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff: Bootstrap starting.
I20260810 03:54:38.743536 28764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cf40db8f527448ebddf019de532025d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:38.744617 29415 tablet_bootstrap.cc:654] T 2cf40db8f527448ebddf019de532025d P 4af47fa8316341f2b1f06b8cd5d43111: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:38.745477 29417 tablet_bootstrap.cc:654] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:38.745947 29415 log.cc:824] T 2cf40db8f527448ebddf019de532025d P 4af47fa8316341f2b1f06b8cd5d43111: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:38.746742 29417 log.cc:824] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:38.748140 29415 tablet_bootstrap.cc:492] T 2cf40db8f527448ebddf019de532025d P 4af47fa8316341f2b1f06b8cd5d43111: No bootstrap required, opened a new log
I20260810 03:54:38.748347 29415 ts_tablet_manager.cc:1397] T 2cf40db8f527448ebddf019de532025d P 4af47fa8316341f2b1f06b8cd5d43111: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 03:54:38.748754 29417 tablet_bootstrap.cc:492] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff: No bootstrap required, opened a new log
I20260810 03:54:38.748956 29417 ts_tablet_manager.cc:1397] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:54:38.750579 29424 tablet_bootstrap.cc:492] T 2cf40db8f527448ebddf019de532025d P 23bd8d6fb9eb4b659de598726841da47: Bootstrap starting.
I20260810 03:54:38.751726 29415 raft_consensus.cc:348] T 2cf40db8f527448ebddf019de532025d P 4af47fa8316341f2b1f06b8cd5d43111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bd8d6fb9eb4b659de598726841da47" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39923 } } peers { permanent_uuid: "4af47fa8316341f2b1f06b8cd5d43111" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 34365 } } peers { permanent_uuid: "2888240ecb554ea2ac709a175997d1ff" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40619 } }
I20260810 03:54:38.752012 29415 raft_consensus.cc:374] T 2cf40db8f527448ebddf019de532025d P 4af47fa8316341f2b1f06b8cd5d43111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:38.752066 29415 raft_consensus.cc:729] T 2cf40db8f527448ebddf019de532025d P 4af47fa8316341f2b1f06b8cd5d43111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4af47fa8316341f2b1f06b8cd5d43111, State: Initialized, Role: FOLLOWER
I20260810 03:54:38.752537 29415 consensus_queue.cc:260] T 2cf40db8f527448ebddf019de532025d P 4af47fa8316341f2b1f06b8cd5d43111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bd8d6fb9eb4b659de598726841da47" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39923 } } peers { permanent_uuid: "4af47fa8316341f2b1f06b8cd5d43111" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 34365 } } peers { permanent_uuid: "2888240ecb554ea2ac709a175997d1ff" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40619 } }
I20260810 03:54:38.752573 29417 raft_consensus.cc:348] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bd8d6fb9eb4b659de598726841da47" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39923 } } peers { permanent_uuid: "4af47fa8316341f2b1f06b8cd5d43111" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 34365 } } peers { permanent_uuid: "2888240ecb554ea2ac709a175997d1ff" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40619 } }
I20260810 03:54:38.752846 29417 raft_consensus.cc:374] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:38.752902 29417 raft_consensus.cc:729] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2888240ecb554ea2ac709a175997d1ff, State: Initialized, Role: FOLLOWER
I20260810 03:54:38.753031 29424 tablet_bootstrap.cc:654] T 2cf40db8f527448ebddf019de532025d P 23bd8d6fb9eb4b659de598726841da47: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:38.753307 29417 consensus_queue.cc:260] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bd8d6fb9eb4b659de598726841da47" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39923 } } peers { permanent_uuid: "4af47fa8316341f2b1f06b8cd5d43111" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 34365 } } peers { permanent_uuid: "2888240ecb554ea2ac709a175997d1ff" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40619 } }
I20260810 03:54:38.753611 29415 ts_tablet_manager.cc:1428] T 2cf40db8f527448ebddf019de532025d P 4af47fa8316341f2b1f06b8cd5d43111: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 03:54:38.753911 29424 log.cc:824] T 2cf40db8f527448ebddf019de532025d P 23bd8d6fb9eb4b659de598726841da47: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:38.753990 29377 heartbeater.cc:503] Master 127.27.153.126:40493 was elected leader, sending a full tablet report...
I20260810 03:54:38.754311 29417 ts_tablet_manager.cc:1428] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 03:54:38.754982 29116 heartbeater.cc:503] Master 127.27.153.126:40493 was elected leader, sending a full tablet report...
I20260810 03:54:38.757592 29424 tablet_bootstrap.cc:492] T 2cf40db8f527448ebddf019de532025d P 23bd8d6fb9eb4b659de598726841da47: No bootstrap required, opened a new log
I20260810 03:54:38.757822 29424 ts_tablet_manager.cc:1397] T 2cf40db8f527448ebddf019de532025d P 23bd8d6fb9eb4b659de598726841da47: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260810 03:54:38.761376 29424 raft_consensus.cc:348] T 2cf40db8f527448ebddf019de532025d P 23bd8d6fb9eb4b659de598726841da47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bd8d6fb9eb4b659de598726841da47" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39923 } } peers { permanent_uuid: "4af47fa8316341f2b1f06b8cd5d43111" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 34365 } } peers { permanent_uuid: "2888240ecb554ea2ac709a175997d1ff" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40619 } }
I20260810 03:54:38.761626 29424 raft_consensus.cc:374] T 2cf40db8f527448ebddf019de532025d P 23bd8d6fb9eb4b659de598726841da47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:38.761662 29424 raft_consensus.cc:729] T 2cf40db8f527448ebddf019de532025d P 23bd8d6fb9eb4b659de598726841da47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23bd8d6fb9eb4b659de598726841da47, State: Initialized, Role: FOLLOWER
I20260810 03:54:38.762084 29424 consensus_queue.cc:260] T 2cf40db8f527448ebddf019de532025d P 23bd8d6fb9eb4b659de598726841da47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bd8d6fb9eb4b659de598726841da47" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39923 } } peers { permanent_uuid: "4af47fa8316341f2b1f06b8cd5d43111" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 34365 } } peers { permanent_uuid: "2888240ecb554ea2ac709a175997d1ff" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40619 } }
I20260810 03:54:38.762818 28875 heartbeater.cc:503] Master 127.27.153.126:40493 was elected leader, sending a full tablet report...
I20260810 03:54:38.763098 29424 ts_tablet_manager.cc:1428] T 2cf40db8f527448ebddf019de532025d P 23bd8d6fb9eb4b659de598726841da47: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260810 03:54:38.931229 29381 tablet.cc:2369] T 2cf40db8f527448ebddf019de532025d P 4af47fa8316341f2b1f06b8cd5d43111: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:54:38.945060 29427 raft_consensus.cc:482] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:54:38.945300 29427 raft_consensus.cc:504] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bd8d6fb9eb4b659de598726841da47" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39923 } } peers { permanent_uuid: "4af47fa8316341f2b1f06b8cd5d43111" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 34365 } } peers { permanent_uuid: "2888240ecb554ea2ac709a175997d1ff" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40619 } }
I20260810 03:54:38.946594 29427 leader_election.cc:288] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23bd8d6fb9eb4b659de598726841da47 (127.27.153.65:39923), 4af47fa8316341f2b1f06b8cd5d43111 (127.27.153.67:34365)
W20260810 03:54:38.947144 29118 tablet.cc:2369] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:54:38.950659 29413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:38.946996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:50499 (local address 127.27.153.67:34365)
0810 03:54:38.947132 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:38.947137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:38.947539 (+ 402us) server_negotiation.cc:407] Connection header received
0810 03:54:38.947661 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:38.947664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:38.947718 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:38.947798 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:38.948070 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.948591 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.949734 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.950140 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.950263 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:38.950326 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:38.950465 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 03:54:38.950534 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:54:38.950670 29122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:38.946856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:39923 (local address 127.27.153.66:33903)
0810 03:54:38.947255 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:38.947271 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:38.947379 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:38.947662 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:38.947664 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:38.947677 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:38.947823 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:38.947830 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.948619 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.948622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:38.949436 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:38.949446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.950182 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.950185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:38.950291 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:38.950325 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 03:54:38.950401 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 03:54:38.950497 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":3}
I20260810 03:54:38.950676 29408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:38.947014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:33903 (local address 127.27.153.65:39923)
0810 03:54:38.947156 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:38.947161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:38.947362 (+ 201us) server_negotiation.cc:407] Connection header received
0810 03:54:38.947522 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:38.947525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:38.947579 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:38.947649 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:38.947965 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.948506 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.949598 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.950055 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.950164 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:38.950214 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:38.950453 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 03:54:38.950542 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 03:54:38.950698 29431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:38.946909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:34365 (local address 127.27.153.66:50499)
0810 03:54:38.947401 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:38.947415 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:38.947549 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:38.947823 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:38.947826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:38.947842 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:38.947967 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:38.947972 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.948717 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.948719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:38.949602 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:38.949613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:38.950256 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:38.950260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:38.950345 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:38.950363 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 03:54:38.950440 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 03:54:38.950542 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":22,"thread_start_us":112,"threads_started":1}
I20260810 03:54:38.951315 28798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf40db8f527448ebddf019de532025d" candidate_uuid: "2888240ecb554ea2ac709a175997d1ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23bd8d6fb9eb4b659de598726841da47" is_pre_election: true
I20260810 03:54:38.951470 29297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf40db8f527448ebddf019de532025d" candidate_uuid: "2888240ecb554ea2ac709a175997d1ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af47fa8316341f2b1f06b8cd5d43111" is_pre_election: true
I20260810 03:54:38.951650 28798 raft_consensus.cc:2454] T 2cf40db8f527448ebddf019de532025d P 23bd8d6fb9eb4b659de598726841da47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2888240ecb554ea2ac709a175997d1ff in term 0.
I20260810 03:54:38.951758 29297 raft_consensus.cc:2454] T 2cf40db8f527448ebddf019de532025d P 4af47fa8316341f2b1f06b8cd5d43111 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2888240ecb554ea2ac709a175997d1ff in term 0.
I20260810 03:54:38.952206 28917 leader_election.cc:302] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23bd8d6fb9eb4b659de598726841da47, 2888240ecb554ea2ac709a175997d1ff; no voters:
I20260810 03:54:38.952451 29427 raft_consensus.cc:2781] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:54:38.952529 29427 raft_consensus.cc:482] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:54:38.952579 29427 raft_consensus.cc:3037] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:38.953615 29427 raft_consensus.cc:504] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bd8d6fb9eb4b659de598726841da47" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39923 } } peers { permanent_uuid: "4af47fa8316341f2b1f06b8cd5d43111" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 34365 } } peers { permanent_uuid: "2888240ecb554ea2ac709a175997d1ff" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40619 } }
I20260810 03:54:38.953958 29427 leader_election.cc:288] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff [CANDIDATE]: Term 1 election: Requested vote from peers 23bd8d6fb9eb4b659de598726841da47 (127.27.153.65:39923), 4af47fa8316341f2b1f06b8cd5d43111 (127.27.153.67:34365)
I20260810 03:54:38.954399 28798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf40db8f527448ebddf019de532025d" candidate_uuid: "2888240ecb554ea2ac709a175997d1ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23bd8d6fb9eb4b659de598726841da47"
I20260810 03:54:38.954475 29297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cf40db8f527448ebddf019de532025d" candidate_uuid: "2888240ecb554ea2ac709a175997d1ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4af47fa8316341f2b1f06b8cd5d43111"
I20260810 03:54:38.954525 28798 raft_consensus.cc:3037] T 2cf40db8f527448ebddf019de532025d P 23bd8d6fb9eb4b659de598726841da47 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:38.954608 29297 raft_consensus.cc:3037] T 2cf40db8f527448ebddf019de532025d P 4af47fa8316341f2b1f06b8cd5d43111 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:38.955662 28798 raft_consensus.cc:2454] T 2cf40db8f527448ebddf019de532025d P 23bd8d6fb9eb4b659de598726841da47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2888240ecb554ea2ac709a175997d1ff in term 1.
I20260810 03:54:38.955708 29297 raft_consensus.cc:2454] T 2cf40db8f527448ebddf019de532025d P 4af47fa8316341f2b1f06b8cd5d43111 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2888240ecb554ea2ac709a175997d1ff in term 1.
I20260810 03:54:38.956017 28917 leader_election.cc:302] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23bd8d6fb9eb4b659de598726841da47, 2888240ecb554ea2ac709a175997d1ff; no voters:
I20260810 03:54:38.956188 29427 raft_consensus.cc:2781] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:54:38.956416 29427 raft_consensus.cc:686] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff [term 1 LEADER]: Becoming Leader. State: Replica: 2888240ecb554ea2ac709a175997d1ff, State: Running, Role: LEADER
I20260810 03:54:38.956794 29427 consensus_queue.cc:237] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bd8d6fb9eb4b659de598726841da47" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39923 } } peers { permanent_uuid: "4af47fa8316341f2b1f06b8cd5d43111" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 34365 } } peers { permanent_uuid: "2888240ecb554ea2ac709a175997d1ff" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40619 } }
I20260810 03:54:38.960022 28531 catalog_manager.cc:5495] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 2888240ecb554ea2ac709a175997d1ff (127.27.153.66). New cstate: current_term: 1 leader_uuid: "2888240ecb554ea2ac709a175997d1ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bd8d6fb9eb4b659de598726841da47" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4af47fa8316341f2b1f06b8cd5d43111" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 34365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2888240ecb554ea2ac709a175997d1ff" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 40619 } health_report { overall_health: HEALTHY } } }
W20260810 03:54:38.967734 28876 tablet.cc:2369] T 2cf40db8f527448ebddf019de532025d P 23bd8d6fb9eb4b659de598726841da47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:54:39.025157 29412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:39.021833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55080 (local address 127.27.153.66:40619)
0810 03:54:39.021960 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:39.021964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:39.021979 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:54:39.022055 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:39.022057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:39.022114 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:39.022210 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:54:39.022407 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:39.022924 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:39.023780 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:39.024001 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:39.024093 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:39.024336 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:54:39.024766 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:54:39.024878 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:39.024938 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 03:54:39.024977 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 03:54:39.031517 29297 raft_consensus.cc:1261] T 2cf40db8f527448ebddf019de532025d P 4af47fa8316341f2b1f06b8cd5d43111 [term 1 FOLLOWER]: Refusing update from remote peer 2888240ecb554ea2ac709a175997d1ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:54:39.031519 28798 raft_consensus.cc:1261] T 2cf40db8f527448ebddf019de532025d P 23bd8d6fb9eb4b659de598726841da47 [term 1 FOLLOWER]: Refusing update from remote peer 2888240ecb554ea2ac709a175997d1ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:54:39.032312 29432 consensus_queue.cc:1035] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "4af47fa8316341f2b1f06b8cd5d43111" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 34365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:54:39.032498 29427 consensus_queue.cc:1035] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "23bd8d6fb9eb4b659de598726841da47" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 39923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:54:39.051424 28531 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:54:39.051637 28531 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:54:39.054293 28531 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eb1999fb9f0149c9a7b79cb78c885d0d: Sending DeleteTablet for 3 replicas of tablet 2cf40db8f527448ebddf019de532025d
I20260810 03:54:39.055388 28764 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cf40db8f527448ebddf019de532025d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:54:39 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:54166
I20260810 03:54:39.055634 28993 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cf40db8f527448ebddf019de532025d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:54:39 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:55070
I20260810 03:54:39.055815 29249 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cf40db8f527448ebddf019de532025d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:54:39 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:56334
I20260810 03:54:39.056192 29452 tablet_replica.cc:330] T 2cf40db8f527448ebddf019de532025d P 23bd8d6fb9eb4b659de598726841da47: stopping tablet replica
I20260810 03:54:39.056362 29457 tablet_replica.cc:330] T 2cf40db8f527448ebddf019de532025d P 4af47fa8316341f2b1f06b8cd5d43111: stopping tablet replica
I20260810 03:54:39.056645 29453 tablet_replica.cc:330] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff: stopping tablet replica
I20260810 03:54:39.056684 29457 raft_consensus.cc:2229] T 2cf40db8f527448ebddf019de532025d P 4af47fa8316341f2b1f06b8cd5d43111 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:54:39.056782 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 28650
I20260810 03:54:39.056951 29453 raft_consensus.cc:2229] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:54:39.057036 29457 raft_consensus.cc:2258] T 2cf40db8f527448ebddf019de532025d P 4af47fa8316341f2b1f06b8cd5d43111 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:54:39.057418 29453 raft_consensus.cc:2258] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:54:39.058822 29457 ts_tablet_manager.cc:1905] T 2cf40db8f527448ebddf019de532025d P 4af47fa8316341f2b1f06b8cd5d43111: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:54:39.062017 29453 ts_tablet_manager.cc:1905] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:54:39.062132 29457 ts_tablet_manager.cc:1918] T 2cf40db8f527448ebddf019de532025d P 4af47fa8316341f2b1f06b8cd5d43111: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:54:39.062256 29457 log.cc:1196] T 2cf40db8f527448ebddf019de532025d P 4af47fa8316341f2b1f06b8cd5d43111: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/2cf40db8f527448ebddf019de532025d
I20260810 03:54:39.062635 29457 ts_tablet_manager.cc:1939] T 2cf40db8f527448ebddf019de532025d P 4af47fa8316341f2b1f06b8cd5d43111: Deleting consensus metadata
I20260810 03:54:39.064272 28506 catalog_manager.cc:4841] TS 4af47fa8316341f2b1f06b8cd5d43111 (127.27.153.67:34365): tablet 2cf40db8f527448ebddf019de532025d (table test-table [id=18fa20cce7ba4355b9c47ae419f119c3]) successfully deleted
I20260810 03:54:39.064689 29453 ts_tablet_manager.cc:1918] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:54:39.064781 29453 log.cc:1196] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/2cf40db8f527448ebddf019de532025d
I20260810 03:54:39.065063 29453 ts_tablet_manager.cc:1939] T 2cf40db8f527448ebddf019de532025d P 2888240ecb554ea2ac709a175997d1ff: Deleting consensus metadata
I20260810 03:54:39.066397 28505 catalog_manager.cc:4841] TS 2888240ecb554ea2ac709a175997d1ff (127.27.153.66:40619): tablet 2cf40db8f527448ebddf019de532025d (table test-table [id=18fa20cce7ba4355b9c47ae419f119c3]) successfully deleted
W20260810 03:54:39.110008 28500 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.153.65:39923 (error 108)
I20260810 03:54:39.110201 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 28883
I20260810 03:54:39.124451 29411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:39.124106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:39923 (local address 127.0.0.1:54170)
0810 03:54:39.124264 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:39.124353 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:39923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 03:54:39.124709 28500 catalog_manager.cc:4584] TS 23bd8d6fb9eb4b659de598726841da47 (127.27.153.65:39923): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2cf40db8f527448ebddf019de532025d: Network error: Client connection negotiation failed: client connection to 127.27.153.65:39923: connect: Connection refused (error 111)
I20260810 03:54:39.130625 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 29130
I20260810 03:54:39.141214 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 28475
2026-08-10T03:54:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2792 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T03:54:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:54:39Z Disabled control of system clock
I20260810 03:54:39.189388 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:39119
--webserver_interface=127.27.153.126
--webserver_port=0
--builtin_ntp_servers=127.27.153.84:40075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:39119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:54:39.303738 29465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:39.304037 29465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:39.304075 29465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:39.308069 29465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:54:39.308135 29465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:39.308149 29465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:39.308164 29465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:54:39.308178 29465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:54:39.313041 29465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:39119
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:39119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:39.314129 29465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:39.315508 29465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:39.324018 29480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:39.324043 29472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:39.324041 29474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:39.324120 29465 server_base.cc:1034] running on GCE node
I20260810 03:54:39.324918 29465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:39.326109 29465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:39.327278 29465 hybrid_clock.cc:648] HybridClock initialized: now 1786334079327264 us; error 37 us; skew 500 ppm
I20260810 03:54:39.329264 29465 webserver.cc:460] Webserver started at http://127.27.153.126:44155/ using document root <none> and password file <none>
I20260810 03:54:39.329862 29465 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:39.329936 29465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:39.330149 29465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:39.332334 29465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "27da3e11befc45b987700ca1299c4bad"
format_stamp: "Formatted at 2026-08-10 03:54:39 on dist-test-slave-pww0"
server_key: "00bc6b9d2fda8b9bb71e4de26a9da960"
server_key_iv: "7d974e4256f073a8dd129d614202d639"
server_key_version: "encryptionkey@0"
I20260810 03:54:39.332885 29465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "27da3e11befc45b987700ca1299c4bad"
format_stamp: "Formatted at 2026-08-10 03:54:39 on dist-test-slave-pww0"
server_key: "00bc6b9d2fda8b9bb71e4de26a9da960"
server_key_iv: "7d974e4256f073a8dd129d614202d639"
server_key_version: "encryptionkey@0"
I20260810 03:54:39.336561 29465 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260810 03:54:39.338996 29491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:39.340296 29465 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 03:54:39.340418 29465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "27da3e11befc45b987700ca1299c4bad"
format_stamp: "Formatted at 2026-08-10 03:54:39 on dist-test-slave-pww0"
server_key: "00bc6b9d2fda8b9bb71e4de26a9da960"
server_key_iv: "7d974e4256f073a8dd129d614202d639"
server_key_version: "encryptionkey@0"
I20260810 03:54:39.340516 29465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:39.357249 29465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:39.358027 29465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:39.358229 29465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:39.365624 29465 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:39119
I20260810 03:54:39.365643 29580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:39119 every 8 connection(s)
I20260810 03:54:39.366716 29465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:54:39.370450 29583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:39.376019 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 29465
I20260810 03:54:39.376179 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:54:39.376488 28261 external_mini_cluster.cc:1428] Setting key 2a9641b705f0a1b19d3467c840b7834a
I20260810 03:54:39.376892 29583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad: Bootstrap starting.
I20260810 03:54:39.379745 29583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:39.380561 29583 log.cc:824] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:39.382588 29583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad: No bootstrap required, opened a new log
I20260810 03:54:39.386404 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:39.378080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40554 (local address 127.27.153.126:39119)
0810 03:54:39.378952 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:39.378969 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:39.379013 (+ 44us) server_negotiation.cc:407] Connection header received
0810 03:54:39.379828 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:39.379857 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:39.380340 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:39.380699 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:39.381167 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:39.382150 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:39.383099 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:39.383636 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:39.383808 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:39.384627 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:39.384632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:39.384646 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:39.385044 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:39.385102 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:39.385122 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:39.385210 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:39.385360 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 03:54:39.385636 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":206,"threads_started":1}
I20260810 03:54:39.386268 29583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27da3e11befc45b987700ca1299c4bad" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39119 } }
I20260810 03:54:39.386497 29583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:39.386546 29583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27da3e11befc45b987700ca1299c4bad, State: Initialized, Role: FOLLOWER
I20260810 03:54:39.387039 29583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27da3e11befc45b987700ca1299c4bad" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39119 } }
I20260810 03:54:39.387187 29583 raft_consensus.cc:388] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:54:39.387239 29583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:54:39.387344 29583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:39.388374 29583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27da3e11befc45b987700ca1299c4bad" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39119 } }
I20260810 03:54:39.388761 29583 leader_election.cc:302] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 27da3e11befc45b987700ca1299c4bad; no voters:
I20260810 03:54:39.389127 29583 leader_election.cc:288] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:54:39.389226 29589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:54:39.389451 29589 raft_consensus.cc:686] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad [term 1 LEADER]: Becoming Leader. State: Replica: 27da3e11befc45b987700ca1299c4bad, State: Running, Role: LEADER
I20260810 03:54:39.389755 29589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27da3e11befc45b987700ca1299c4bad" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39119 } }
I20260810 03:54:39.390369 29583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:54:39.391537 29593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 27da3e11befc45b987700ca1299c4bad. Latest consensus state: current_term: 1 leader_uuid: "27da3e11befc45b987700ca1299c4bad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27da3e11befc45b987700ca1299c4bad" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39119 } } }
I20260810 03:54:39.391672 29593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:39.391923 29592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "27da3e11befc45b987700ca1299c4bad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "27da3e11befc45b987700ca1299c4bad" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39119 } } }
I20260810 03:54:39.392019 29592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:39.392145 29609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:54:39.395344 29609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:54:39.401057 29609 catalog_manager.cc:1347] Generated new cluster ID: 1194d2f221554e95976ed36a148d8994
I20260810 03:54:39.401144 29609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:54:39.409096 29609 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:54:39.410050 29609 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:54:39.415863 29609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad: Generated new TSK 0
I20260810 03:54:39.416596 29609 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:54:39.420114 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:39119
--builtin_ntp_servers=127.27.153.84:40075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:54:39.564751 29628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:39.565052 29628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:39.565106 29628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:39.569242 29628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:39.569339 29628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:39.569402 29628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:54:39.574288 29628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:39119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:39.575408 29628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:39.576820 29628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:39.586798 29636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:39.586818 29635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:39.586818 29642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:39.587028 29628 server_base.cc:1034] running on GCE node
I20260810 03:54:39.587555 29628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:39.588240 29628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:39.589466 29628 hybrid_clock.cc:648] HybridClock initialized: now 1786334079589442 us; error 37 us; skew 500 ppm
I20260810 03:54:39.591629 29628 webserver.cc:460] Webserver started at http://127.27.153.65:33107/ using document root <none> and password file <none>
I20260810 03:54:39.592250 29628 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:39.592317 29628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:39.592526 29628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:39.594753 29628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "18cd51281d524734b7488386bc3dbae1"
format_stamp: "Formatted at 2026-08-10 03:54:39 on dist-test-slave-pww0"
server_key: "f6672505a5ee87f736138e96520159b0"
server_key_iv: "6b9f944870ceaff0822fd310c91b20d7"
server_key_version: "encryptionkey@0"
I20260810 03:54:39.595325 29628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "18cd51281d524734b7488386bc3dbae1"
format_stamp: "Formatted at 2026-08-10 03:54:39 on dist-test-slave-pww0"
server_key: "f6672505a5ee87f736138e96520159b0"
server_key_iv: "6b9f944870ceaff0822fd310c91b20d7"
server_key_version: "encryptionkey@0"
I20260810 03:54:39.599274 29628 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 03:54:39.601933 29657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:39.603222 29628 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 03:54:39.603359 29628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "18cd51281d524734b7488386bc3dbae1"
format_stamp: "Formatted at 2026-08-10 03:54:39 on dist-test-slave-pww0"
server_key: "f6672505a5ee87f736138e96520159b0"
server_key_iv: "6b9f944870ceaff0822fd310c91b20d7"
server_key_version: "encryptionkey@0"
I20260810 03:54:39.603500 29628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:39.621141 29628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:39.621898 29628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:39.622090 29628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:39.622722 29628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:54:39.623795 29628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:54:39.623845 29628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:39.623901 29628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:54:39.623929 29628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:39.632973 29628 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:43241
I20260810 03:54:39.633005 29877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:43241 every 8 connection(s)
I20260810 03:54:39.633939 29628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:54:39.640736 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 29628
I20260810 03:54:39.640872 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:54:39.641146 28261 external_mini_cluster.cc:1428] Setting key dc4d0f2f8fc4addd1c39a4bc782b739a
I20260810 03:54:39.643411 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:39119
--builtin_ntp_servers=127.27.153.84:40075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:54:39.643805 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:39.635887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:34607 (local address 127.27.153.126:39119)
0810 03:54:39.636081 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:39.636087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:39.637205 (+ 1118us) server_negotiation.cc:407] Connection header received
0810 03:54:39.638611 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:39.638614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:39.638669 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:39.638754 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:39.639749 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:39.640312 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:39.641550 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:39.641793 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:39.641882 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:39.643204 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:39.643208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:39.643210 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:39.643223 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:39.643246 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:39.643254 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:39.643321 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:39.643557 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 03:54:39.643655 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 03:54:39.644966 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:39.636335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39119 (local address 127.27.153.65:34607)
0810 03:54:39.637040 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:39.637082 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:39.638368 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:39.638928 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:39.638935 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:39.639023 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:54:39.639584 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:39.639601 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:39.640422 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:39.640425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:39.641422 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:39.641434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:39.641959 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:39.641963 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:39.642118 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:39.642856 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:54:39.642858 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:54:39.642918 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:54:39.643013 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:54:39.643344 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:54:39.643355 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:54:39.643362 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:54:39.643557 (+ 195us) client_negotiation.cc:240] Negotiation successful
0810 03:54:39.643889 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":126,"threads_started":1}
I20260810 03:54:39.646440 29878 heartbeater.cc:347] Connected to a master server at 127.27.153.126:39119
I20260810 03:54:39.646692 29878 heartbeater.cc:464] Registering TS with master...
I20260810 03:54:39.647262 29878 heartbeater.cc:511] Master 127.27.153.126:39119 requested a full tablet report, sending...
I20260810 03:54:39.648957 29522 ts_manager.cc:194] Registered new tserver with Master: 18cd51281d524734b7488386bc3dbae1 (127.27.153.65:43241)
I20260810 03:54:39.650161 29522 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.153.65:34607
W20260810 03:54:39.762326 29882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:39.762647 29882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:39.762697 29882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:39.767040 29882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:39.767129 29882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:39.767194 29882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:54:39.772293 29882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:39119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:39.773489 29882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:39.774961 29882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:39.784844 29889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:39.784844 29890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:39.785064 29882 server_base.cc:1034] running on GCE node
W20260810 03:54:39.785202 29898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:39.785604 29882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:39.786265 29882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:39.787448 29882 hybrid_clock.cc:648] HybridClock initialized: now 1786334079787424 us; error 34 us; skew 500 ppm
I20260810 03:54:39.789397 29882 webserver.cc:460] Webserver started at http://127.27.153.66:36855/ using document root <none> and password file <none>
I20260810 03:54:39.790016 29882 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:39.790091 29882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:39.790328 29882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:39.792645 29882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "1b941f423f0046e882a475818c14bf9e"
format_stamp: "Formatted at 2026-08-10 03:54:39 on dist-test-slave-pww0"
server_key: "dd6c7aff3fd7a4daec22fdfa329ac1ad"
server_key_iv: "e8c1b0597ed7fe9909af3f85b855523d"
server_key_version: "encryptionkey@0"
I20260810 03:54:39.793220 29882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "1b941f423f0046e882a475818c14bf9e"
format_stamp: "Formatted at 2026-08-10 03:54:39 on dist-test-slave-pww0"
server_key: "dd6c7aff3fd7a4daec22fdfa329ac1ad"
server_key_iv: "e8c1b0597ed7fe9909af3f85b855523d"
server_key_version: "encryptionkey@0"
I20260810 03:54:39.796976 29882 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:54:39.799418 29911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:39.800680 29882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:54:39.800802 29882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "1b941f423f0046e882a475818c14bf9e"
format_stamp: "Formatted at 2026-08-10 03:54:39 on dist-test-slave-pww0"
server_key: "dd6c7aff3fd7a4daec22fdfa329ac1ad"
server_key_iv: "e8c1b0597ed7fe9909af3f85b855523d"
server_key_version: "encryptionkey@0"
I20260810 03:54:39.800915 29882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:39.812812 29882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:39.813498 29882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:39.813683 29882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:39.814280 29882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:54:39.815248 29882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:54:39.815297 29882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:39.815351 29882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:54:39.815375 29882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:39.824082 29882 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:45039
I20260810 03:54:39.824113 30126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:45039 every 8 connection(s)
I20260810 03:54:39.825021 29882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:54:39.832105 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 29882
I20260810 03:54:39.832257 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:54:39.832582 28261 external_mini_cluster.cc:1428] Setting key f74650d515fd8ef0c608d7d018b0eb87
I20260810 03:54:39.834724 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:39.826835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:57467 (local address 127.27.153.126:39119)
0810 03:54:39.826981 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:39.826985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:39.828124 (+ 1139us) server_negotiation.cc:407] Connection header received
0810 03:54:39.829551 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:39.829554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:39.829617 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:39.829695 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:39.830675 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:39.831217 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:39.832726 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:39.832971 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:39.833095 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:39.834262 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:39.834265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:39.834267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:39.834278 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:39.834299 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:39.834306 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:39.834363 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:39.834530 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 03:54:39.834584 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:54:39.834837 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:39119
--builtin_ntp_servers=127.27.153.84:40075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:54:39.835956 30132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:39.827267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39119 (local address 127.27.153.66:57467)
0810 03:54:39.827967 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:39.828010 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:39.829316 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:39.829896 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:39.829903 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:39.829994 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:54:39.830525 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:39.830539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:39.831350 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:39.831353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:39.832556 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:39.832570 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:39.833134 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:39.833137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:39.833337 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:39.833983 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:54:39.833984 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:54:39.834026 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:54:39.834105 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:54:39.834383 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:54:39.834392 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:54:39.834397 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 03:54:39.834528 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 03:54:39.834850 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":162,"threads_started":1}
I20260810 03:54:39.837093 30127 heartbeater.cc:347] Connected to a master server at 127.27.153.126:39119
I20260810 03:54:39.837344 30127 heartbeater.cc:464] Registering TS with master...
I20260810 03:54:39.837877 30127 heartbeater.cc:511] Master 127.27.153.126:39119 requested a full tablet report, sending...
I20260810 03:54:39.838950 29522 ts_manager.cc:194] Registered new tserver with Master: 1b941f423f0046e882a475818c14bf9e (127.27.153.66:45039)
I20260810 03:54:39.839560 29522 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.153.66:57467
W20260810 03:54:39.951610 30134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:39.951896 30134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:39.951937 30134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:39.956110 30134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:39.956195 30134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:39.956259 30134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:54:39.961336 30134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:39119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:39.962531 30134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:39.964004 30134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:39.974319 30141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:39.974313 30146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:39.974313 30143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:39.974365 30134 server_base.cc:1034] running on GCE node
I20260810 03:54:39.974967 30134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:39.975711 30134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:39.976908 30134 hybrid_clock.cc:648] HybridClock initialized: now 1786334079976884 us; error 34 us; skew 500 ppm
I20260810 03:54:39.978897 30134 webserver.cc:460] Webserver started at http://127.27.153.67:46149/ using document root <none> and password file <none>
I20260810 03:54:39.979557 30134 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:39.979641 30134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:39.979871 30134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:39.982090 30134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "c8d6ed154f37483d9a7fe2e615a7d29d"
format_stamp: "Formatted at 2026-08-10 03:54:39 on dist-test-slave-pww0"
server_key: "10eeba36d6594686db90a5bebaa9afd3"
server_key_iv: "1c54e885908e30e27b1fb55c02b3fa5c"
server_key_version: "encryptionkey@0"
I20260810 03:54:39.982694 30134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "c8d6ed154f37483d9a7fe2e615a7d29d"
format_stamp: "Formatted at 2026-08-10 03:54:39 on dist-test-slave-pww0"
server_key: "10eeba36d6594686db90a5bebaa9afd3"
server_key_iv: "1c54e885908e30e27b1fb55c02b3fa5c"
server_key_version: "encryptionkey@0"
I20260810 03:54:39.986649 30134 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:54:39.989400 30157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:39.990833 30134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:54:39.990975 30134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "c8d6ed154f37483d9a7fe2e615a7d29d"
format_stamp: "Formatted at 2026-08-10 03:54:39 on dist-test-slave-pww0"
server_key: "10eeba36d6594686db90a5bebaa9afd3"
server_key_iv: "1c54e885908e30e27b1fb55c02b3fa5c"
server_key_version: "encryptionkey@0"
I20260810 03:54:39.991091 30134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:40.010893 30134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:40.011732 30134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:40.011929 30134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:40.012578 30134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:54:40.013568 30134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:54:40.013618 30134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:40.013674 30134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:54:40.013703 30134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:40.022385 30134 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:36971
I20260810 03:54:40.022435 30358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:36971 every 8 connection(s)
I20260810 03:54:40.023344 30134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:54:40.032856 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 30134
I20260810 03:54:40.032933 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:40.025293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:47281 (local address 127.27.153.126:39119)
0810 03:54:40.025444 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:40.025449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:40.026671 (+ 1222us) server_negotiation.cc:407] Connection header received
0810 03:54:40.028111 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:40.028114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:40.028166 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:40.028250 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:40.029161 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.029685 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.030903 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.031133 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.031222 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:40.032411 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:40.032414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:40.032416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:40.032436 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:40.032462 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:40.032471 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:40.032531 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:40.032678 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 03:54:40.032771 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:54:40.032987 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:54:40.033284 28261 external_mini_cluster.cc:1428] Setting key 3ac4901cfc736cacf1ba8f94908385f9
I20260810 03:54:40.033928 30364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:40.025700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:39119 (local address 127.27.153.67:47281)
0810 03:54:40.026472 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:40.026518 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:40.027892 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:40.028412 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:40.028419 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:40.028509 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:54:40.029015 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:40.029030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.029827 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.029830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:40.030772 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:40.030784 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.031280 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.031283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:40.031435 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:40.032113 (+ 678us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:54:40.032114 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:54:40.032159 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:54:40.032231 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:54:40.032556 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:54:40.032562 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:54:40.032566 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 03:54:40.032660 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 03:54:40.032911 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":139,"threads_started":1}
I20260810 03:54:40.035195 30359 heartbeater.cc:347] Connected to a master server at 127.27.153.126:39119
I20260810 03:54:40.035480 30359 heartbeater.cc:464] Registering TS with master...
I20260810 03:54:40.036092 30359 heartbeater.cc:511] Master 127.27.153.126:39119 requested a full tablet report, sending...
I20260810 03:54:40.038267 29522 ts_manager.cc:194] Registered new tserver with Master: c8d6ed154f37483d9a7fe2e615a7d29d (127.27.153.67:36971)
I20260810 03:54:40.038844 29522 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.153.67:47281
I20260810 03:54:40.047991 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:54:40.054258 29587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:40.050133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40562 (local address 127.27.153.126:39119)
0810 03:54:40.050327 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:40.050333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:40.050480 (+ 147us) server_negotiation.cc:407] Connection header received
0810 03:54:40.050616 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:40.050619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:40.050664 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:40.050735 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:40.051035 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.051611 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.052464 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.052685 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.052774 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:40.053729 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:40.053732 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:40.053735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:40.053748 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:40.053769 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:40.053777 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:40.053890 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:40.054045 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 03:54:40.054105 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 03:54:40.057394 29522 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:54:40.060457 29522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:54:40.075270 30382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:40.069728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34042 (local address 127.27.153.65:43241)
0810 03:54:40.070058 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:40.070071 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:40.070092 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:54:40.070172 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:40.070178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:40.070404 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:40.070930 (+ 526us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:40.071072 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.072214 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.074014 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.074574 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.074676 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:40.074891 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:40.075003 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 03:54:40.075111 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":104,"threads_started":1}
I20260810 03:54:40.075268 30378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:40.069001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:43241 (local address 127.0.0.1:34042)
0810 03:54:40.069422 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:40.069483 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:40.069633 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:40.070584 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:40.070601 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:40.070625 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:40.070779 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:40.070786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.072348 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.072352 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:40.073870 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:40.073881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.074692 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.074695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:40.074816 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:40.074856 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 03:54:40.074971 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 03:54:40.075106 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":120,"thread_start_us":84,"threads_started":1}
I20260810 03:54:40.075647 30381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:40.069322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51084 (local address 127.27.153.66:45039)
0810 03:54:40.069743 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:40.069755 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:40.069993 (+ 238us) server_negotiation.cc:407] Connection header received
0810 03:54:40.070219 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:40.070224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:40.070404 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:40.070541 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:40.070922 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.072177 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.073854 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.074647 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.074756 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:40.074953 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:40.075354 (+ 401us) server_negotiation.cc:299] Negotiation successful
0810 03:54:40.075488 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":100,"threads_started":1}
I20260810 03:54:40.075874 30384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:40.070199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40560 (local address 127.27.153.67:36971)
0810 03:54:40.070663 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:40.070675 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:40.070922 (+ 247us) server_negotiation.cc:407] Connection header received
0810 03:54:40.071112 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:40.071117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:40.071268 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:40.071398 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:40.071629 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.072401 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.074002 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.074720 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.074816 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:40.074957 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:40.075538 (+ 581us) server_negotiation.cc:299] Negotiation successful
0810 03:54:40.075632 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":152,"threads_started":1}
I20260810 03:54:40.076016 30379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:40.069045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:45039 (local address 127.0.0.1:51084)
0810 03:54:40.069890 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:40.069903 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:40.070067 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:40.070720 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:40.070723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:40.070738 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:40.070802 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:40.070808 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.072299 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.072302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:40.073677 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:40.073688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.075018 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.075023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:40.075168 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:40.075210 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 03:54:40.075805 (+ 595us) client_negotiation.cc:240] Negotiation successful
0810 03:54:40.075884 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"mutex_wait_us":174,"thread_start_us":98,"threads_started":1}
I20260810 03:54:40.076395 30383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:40.069795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:36971 (local address 127.0.0.1:40560)
0810 03:54:40.070828 (+ 1033us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:40.070840 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:40.070983 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:40.071399 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:40.071402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:40.071418 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:40.071526 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:40.071533 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.072510 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.072513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:40.073869 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:40.073881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.075018 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.075022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:40.075164 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:40.075216 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 03:54:40.075282 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 03:54:40.076265 (+ 983us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":80,"thread_start_us":89,"threads_started":1}
I20260810 03:54:40.078778 30241 tablet_service.cc:1467] Processing CreateTablet for tablet 4c0d3fe39fef446d8c2d0f88c403baaf (DEFAULT_TABLE table=test-table [id=d33053f33838491caffd01d341ad5f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:54:40.078675 29744 tablet_service.cc:1467] Processing CreateTablet for tablet 4c0d3fe39fef446d8c2d0f88c403baaf (DEFAULT_TABLE table=test-table [id=d33053f33838491caffd01d341ad5f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:54:40.078675 29999 tablet_service.cc:1467] Processing CreateTablet for tablet 4c0d3fe39fef446d8c2d0f88c403baaf (DEFAULT_TABLE table=test-table [id=d33053f33838491caffd01d341ad5f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:54:40.079809 30241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c0d3fe39fef446d8c2d0f88c403baaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:40.079828 29744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c0d3fe39fef446d8c2d0f88c403baaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:40.079828 29999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c0d3fe39fef446d8c2d0f88c403baaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:40.087181 30387 tablet_bootstrap.cc:492] T 4c0d3fe39fef446d8c2d0f88c403baaf P 18cd51281d524734b7488386bc3dbae1: Bootstrap starting.
I20260810 03:54:40.089304 30387 tablet_bootstrap.cc:654] T 4c0d3fe39fef446d8c2d0f88c403baaf P 18cd51281d524734b7488386bc3dbae1: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:40.089833 30385 tablet_bootstrap.cc:492] T 4c0d3fe39fef446d8c2d0f88c403baaf P c8d6ed154f37483d9a7fe2e615a7d29d: Bootstrap starting.
I20260810 03:54:40.090113 30387 log.cc:824] T 4c0d3fe39fef446d8c2d0f88c403baaf P 18cd51281d524734b7488386bc3dbae1: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:40.090277 30388 tablet_bootstrap.cc:492] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e: Bootstrap starting.
I20260810 03:54:40.092227 30385 tablet_bootstrap.cc:654] T 4c0d3fe39fef446d8c2d0f88c403baaf P c8d6ed154f37483d9a7fe2e615a7d29d: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:40.092526 30388 tablet_bootstrap.cc:654] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:40.092795 30387 tablet_bootstrap.cc:492] T 4c0d3fe39fef446d8c2d0f88c403baaf P 18cd51281d524734b7488386bc3dbae1: No bootstrap required, opened a new log
I20260810 03:54:40.093004 30387 ts_tablet_manager.cc:1397] T 4c0d3fe39fef446d8c2d0f88c403baaf P 18cd51281d524734b7488386bc3dbae1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 03:54:40.093181 30385 log.cc:824] T 4c0d3fe39fef446d8c2d0f88c403baaf P c8d6ed154f37483d9a7fe2e615a7d29d: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:40.093336 30388 log.cc:824] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:40.095284 30388 tablet_bootstrap.cc:492] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e: No bootstrap required, opened a new log
I20260810 03:54:40.095378 30385 tablet_bootstrap.cc:492] T 4c0d3fe39fef446d8c2d0f88c403baaf P c8d6ed154f37483d9a7fe2e615a7d29d: No bootstrap required, opened a new log
I20260810 03:54:40.095520 30388 ts_tablet_manager.cc:1397] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260810 03:54:40.095607 30385 ts_tablet_manager.cc:1397] T 4c0d3fe39fef446d8c2d0f88c403baaf P c8d6ed154f37483d9a7fe2e615a7d29d: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260810 03:54:40.097455 30387 raft_consensus.cc:348] T 4c0d3fe39fef446d8c2d0f88c403baaf P 18cd51281d524734b7488386bc3dbae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cd51281d524734b7488386bc3dbae1" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43241 } } peers { permanent_uuid: "1b941f423f0046e882a475818c14bf9e" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 45039 } } peers { permanent_uuid: "c8d6ed154f37483d9a7fe2e615a7d29d" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 36971 } }
I20260810 03:54:40.097759 30387 raft_consensus.cc:374] T 4c0d3fe39fef446d8c2d0f88c403baaf P 18cd51281d524734b7488386bc3dbae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:40.097821 30387 raft_consensus.cc:729] T 4c0d3fe39fef446d8c2d0f88c403baaf P 18cd51281d524734b7488386bc3dbae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cd51281d524734b7488386bc3dbae1, State: Initialized, Role: FOLLOWER
I20260810 03:54:40.098331 30387 consensus_queue.cc:260] T 4c0d3fe39fef446d8c2d0f88c403baaf P 18cd51281d524734b7488386bc3dbae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cd51281d524734b7488386bc3dbae1" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43241 } } peers { permanent_uuid: "1b941f423f0046e882a475818c14bf9e" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 45039 } } peers { permanent_uuid: "c8d6ed154f37483d9a7fe2e615a7d29d" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 36971 } }
I20260810 03:54:40.099107 29878 heartbeater.cc:503] Master 127.27.153.126:39119 was elected leader, sending a full tablet report...
I20260810 03:54:40.099103 30388 raft_consensus.cc:348] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cd51281d524734b7488386bc3dbae1" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43241 } } peers { permanent_uuid: "1b941f423f0046e882a475818c14bf9e" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 45039 } } peers { permanent_uuid: "c8d6ed154f37483d9a7fe2e615a7d29d" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 36971 } }
I20260810 03:54:40.099392 30388 raft_consensus.cc:374] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:40.099452 30387 ts_tablet_manager.cc:1428] T 4c0d3fe39fef446d8c2d0f88c403baaf P 18cd51281d524734b7488386bc3dbae1: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 03:54:40.099478 30388 raft_consensus.cc:729] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b941f423f0046e882a475818c14bf9e, State: Initialized, Role: FOLLOWER
I20260810 03:54:40.099916 30388 consensus_queue.cc:260] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cd51281d524734b7488386bc3dbae1" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43241 } } peers { permanent_uuid: "1b941f423f0046e882a475818c14bf9e" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 45039 } } peers { permanent_uuid: "c8d6ed154f37483d9a7fe2e615a7d29d" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 36971 } }
I20260810 03:54:40.100502 30385 raft_consensus.cc:348] T 4c0d3fe39fef446d8c2d0f88c403baaf P c8d6ed154f37483d9a7fe2e615a7d29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cd51281d524734b7488386bc3dbae1" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43241 } } peers { permanent_uuid: "1b941f423f0046e882a475818c14bf9e" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 45039 } } peers { permanent_uuid: "c8d6ed154f37483d9a7fe2e615a7d29d" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 36971 } }
I20260810 03:54:40.100762 30385 raft_consensus.cc:374] T 4c0d3fe39fef446d8c2d0f88c403baaf P c8d6ed154f37483d9a7fe2e615a7d29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:40.100828 30385 raft_consensus.cc:729] T 4c0d3fe39fef446d8c2d0f88c403baaf P c8d6ed154f37483d9a7fe2e615a7d29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d6ed154f37483d9a7fe2e615a7d29d, State: Initialized, Role: FOLLOWER
I20260810 03:54:40.101042 30127 heartbeater.cc:503] Master 127.27.153.126:39119 was elected leader, sending a full tablet report...
I20260810 03:54:40.101228 30385 consensus_queue.cc:260] T 4c0d3fe39fef446d8c2d0f88c403baaf P c8d6ed154f37483d9a7fe2e615a7d29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cd51281d524734b7488386bc3dbae1" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43241 } } peers { permanent_uuid: "1b941f423f0046e882a475818c14bf9e" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 45039 } } peers { permanent_uuid: "c8d6ed154f37483d9a7fe2e615a7d29d" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 36971 } }
I20260810 03:54:40.101541 30388 ts_tablet_manager.cc:1428] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:54:40.102160 30385 ts_tablet_manager.cc:1428] T 4c0d3fe39fef446d8c2d0f88c403baaf P c8d6ed154f37483d9a7fe2e615a7d29d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:54:40.102356 30359 heartbeater.cc:503] Master 127.27.153.126:39119 was elected leader, sending a full tablet report...
W20260810 03:54:40.135522 29879 tablet.cc:2369] T 4c0d3fe39fef446d8c2d0f88c403baaf P 18cd51281d524734b7488386bc3dbae1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:54:40.188557 30393 raft_consensus.cc:482] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:54:40.188779 30393 raft_consensus.cc:504] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cd51281d524734b7488386bc3dbae1" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43241 } } peers { permanent_uuid: "1b941f423f0046e882a475818c14bf9e" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 45039 } } peers { permanent_uuid: "c8d6ed154f37483d9a7fe2e615a7d29d" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 36971 } }
I20260810 03:54:40.189924 30393 leader_election.cc:288] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cd51281d524734b7488386bc3dbae1 (127.27.153.65:43241), c8d6ed154f37483d9a7fe2e615a7d29d (127.27.153.67:36971)
I20260810 03:54:40.194216 30132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:40.190227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:43241 (local address 127.27.153.66:59153)
0810 03:54:40.190518 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:40.190559 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:40.190666 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:40.190930 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:40.190933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:40.190951 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:40.191093 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:40.191101 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.191908 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.191912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:40.193129 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:40.193140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.193800 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.193803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:40.193899 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:40.193918 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 03:54:40.193985 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 03:54:40.194070 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":132}
I20260810 03:54:40.194235 30382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:40.190338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:59153 (local address 127.27.153.65:43241)
0810 03:54:40.190613 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:40.190617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:40.190634 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:54:40.190773 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:40.190776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:40.190822 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:40.190902 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:40.191236 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.191801 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.193283 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.193677 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.193788 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:40.193840 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:40.194011 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 03:54:40.194085 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260810 03:54:40.194227 30384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:40.190353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:59563 (local address 127.27.153.67:36971)
0810 03:54:40.190486 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:40.190491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:40.190838 (+ 347us) server_negotiation.cc:407] Connection header received
0810 03:54:40.190924 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:40.190926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:40.190986 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:40.191051 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:40.191340 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.191866 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.193303 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.193708 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.193824 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:40.193875 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:40.194034 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 03:54:40.194108 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 03:54:40.194257 30399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:40.190269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:36971 (local address 127.27.153.66:59563)
0810 03:54:40.190738 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:40.190750 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:40.190839 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:40.191076 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:40.191079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:40.191095 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:40.191228 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:40.191234 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.191966 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.191969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:40.193135 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:40.193146 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.193825 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.193828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:40.193925 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:40.193947 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 03:54:40.194012 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 03:54:40.194090 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":38,"thread_start_us":134,"threads_started":1}
I20260810 03:54:40.195186 29781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0d3fe39fef446d8c2d0f88c403baaf" candidate_uuid: "1b941f423f0046e882a475818c14bf9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cd51281d524734b7488386bc3dbae1" is_pre_election: true
I20260810 03:54:40.195564 29781 raft_consensus.cc:2454] T 4c0d3fe39fef446d8c2d0f88c403baaf P 18cd51281d524734b7488386bc3dbae1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b941f423f0046e882a475818c14bf9e in term 0.
I20260810 03:54:40.195494 30275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0d3fe39fef446d8c2d0f88c403baaf" candidate_uuid: "1b941f423f0046e882a475818c14bf9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d6ed154f37483d9a7fe2e615a7d29d" is_pre_election: true
I20260810 03:54:40.195832 30275 raft_consensus.cc:2454] T 4c0d3fe39fef446d8c2d0f88c403baaf P c8d6ed154f37483d9a7fe2e615a7d29d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b941f423f0046e882a475818c14bf9e in term 0.
I20260810 03:54:40.196115 29918 leader_election.cc:302] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cd51281d524734b7488386bc3dbae1, 1b941f423f0046e882a475818c14bf9e; no voters:
I20260810 03:54:40.196434 30393 raft_consensus.cc:2781] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:54:40.196513 30393 raft_consensus.cc:482] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:54:40.196563 30393 raft_consensus.cc:3037] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:40.197580 30393 raft_consensus.cc:504] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cd51281d524734b7488386bc3dbae1" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43241 } } peers { permanent_uuid: "1b941f423f0046e882a475818c14bf9e" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 45039 } } peers { permanent_uuid: "c8d6ed154f37483d9a7fe2e615a7d29d" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 36971 } }
I20260810 03:54:40.197925 30393 leader_election.cc:288] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e [CANDIDATE]: Term 1 election: Requested vote from peers 18cd51281d524734b7488386bc3dbae1 (127.27.153.65:43241), c8d6ed154f37483d9a7fe2e615a7d29d (127.27.153.67:36971)
I20260810 03:54:40.198297 29781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0d3fe39fef446d8c2d0f88c403baaf" candidate_uuid: "1b941f423f0046e882a475818c14bf9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cd51281d524734b7488386bc3dbae1"
I20260810 03:54:40.198436 29781 raft_consensus.cc:3037] T 4c0d3fe39fef446d8c2d0f88c403baaf P 18cd51281d524734b7488386bc3dbae1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:40.198483 30275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c0d3fe39fef446d8c2d0f88c403baaf" candidate_uuid: "1b941f423f0046e882a475818c14bf9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d6ed154f37483d9a7fe2e615a7d29d"
I20260810 03:54:40.198626 30275 raft_consensus.cc:3037] T 4c0d3fe39fef446d8c2d0f88c403baaf P c8d6ed154f37483d9a7fe2e615a7d29d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:40.199542 29781 raft_consensus.cc:2454] T 4c0d3fe39fef446d8c2d0f88c403baaf P 18cd51281d524734b7488386bc3dbae1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b941f423f0046e882a475818c14bf9e in term 1.
I20260810 03:54:40.199813 30275 raft_consensus.cc:2454] T 4c0d3fe39fef446d8c2d0f88c403baaf P c8d6ed154f37483d9a7fe2e615a7d29d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b941f423f0046e882a475818c14bf9e in term 1.
I20260810 03:54:40.199890 29918 leader_election.cc:302] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cd51281d524734b7488386bc3dbae1, 1b941f423f0046e882a475818c14bf9e; no voters:
I20260810 03:54:40.200067 30393 raft_consensus.cc:2781] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:54:40.200318 30393 raft_consensus.cc:686] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e [term 1 LEADER]: Becoming Leader. State: Replica: 1b941f423f0046e882a475818c14bf9e, State: Running, Role: LEADER
I20260810 03:54:40.200644 30393 consensus_queue.cc:237] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cd51281d524734b7488386bc3dbae1" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43241 } } peers { permanent_uuid: "1b941f423f0046e882a475818c14bf9e" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 45039 } } peers { permanent_uuid: "c8d6ed154f37483d9a7fe2e615a7d29d" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 36971 } }
I20260810 03:54:40.204037 29520 catalog_manager.cc:5495] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b941f423f0046e882a475818c14bf9e (127.27.153.66). New cstate: current_term: 1 leader_uuid: "1b941f423f0046e882a475818c14bf9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cd51281d524734b7488386bc3dbae1" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b941f423f0046e882a475818c14bf9e" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 45039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8d6ed154f37483d9a7fe2e615a7d29d" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 36971 } health_report { overall_health: UNKNOWN } } }
I20260810 03:54:40.232216 30381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:40.228572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51092 (local address 127.27.153.66:45039)
0810 03:54:40.228710 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:40.228714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:40.228731 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:54:40.228835 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:40.228837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:40.228895 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:40.228985 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:54:40.229291 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.229814 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.230713 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:40.230919 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:40.230995 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:40.231293 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:54:40.231750 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:54:40.231864 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:40.231981 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 03:54:40.232020 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 03:54:40.238392 29781 raft_consensus.cc:1261] T 4c0d3fe39fef446d8c2d0f88c403baaf P 18cd51281d524734b7488386bc3dbae1 [term 1 FOLLOWER]: Refusing update from remote peer 1b941f423f0046e882a475818c14bf9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:54:40.238392 30275 raft_consensus.cc:1261] T 4c0d3fe39fef446d8c2d0f88c403baaf P c8d6ed154f37483d9a7fe2e615a7d29d [term 1 FOLLOWER]: Refusing update from remote peer 1b941f423f0046e882a475818c14bf9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:54:40.239061 30393 consensus_queue.cc:1035] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cd51281d524734b7488386bc3dbae1" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:54:40.239280 30400 consensus_queue.cc:1035] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d6ed154f37483d9a7fe2e615a7d29d" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 36971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:54:40.247736 30407 mvcc.cc:205] Tried to move back new op lower bound from 7316824392649064448 to 7316824392504455168. Current Snapshot: MvccSnapshot[applied={T|T < 7316824392649064448}]
I20260810 03:54:40.250823 30408 mvcc.cc:205] Tried to move back new op lower bound from 7316824392649064448 to 7316824392504455168. Current Snapshot: MvccSnapshot[applied={T|T < 7316824392649064448 or (T in {7316824392649064448})}]
I20260810 03:54:40.257510 29520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:54:40.257692 29520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:54:40.260334 29520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 27da3e11befc45b987700ca1299c4bad: Sending DeleteTablet for 3 replicas of tablet 4c0d3fe39fef446d8c2d0f88c403baaf
I20260810 03:54:40.261373 29744 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0d3fe39fef446d8c2d0f88c403baaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:54:40 UTC) from {username='slave'} at 127.0.0.1:34042
I20260810 03:54:40.261469 30241 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0d3fe39fef446d8c2d0f88c403baaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:54:40 UTC) from {username='slave'} at 127.0.0.1:40560
I20260810 03:54:40.261473 29999 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c0d3fe39fef446d8c2d0f88c403baaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:54:40 UTC) from {username='slave'} at 127.0.0.1:51084
I20260810 03:54:40.262063 30424 tablet_replica.cc:330] T 4c0d3fe39fef446d8c2d0f88c403baaf P c8d6ed154f37483d9a7fe2e615a7d29d: stopping tablet replica
I20260810 03:54:40.262207 30425 tablet_replica.cc:330] T 4c0d3fe39fef446d8c2d0f88c403baaf P 18cd51281d524734b7488386bc3dbae1: stopping tablet replica
I20260810 03:54:40.262451 30424 raft_consensus.cc:2229] T 4c0d3fe39fef446d8c2d0f88c403baaf P c8d6ed154f37483d9a7fe2e615a7d29d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:54:40.262612 30425 raft_consensus.cc:2229] T 4c0d3fe39fef446d8c2d0f88c403baaf P 18cd51281d524734b7488386bc3dbae1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:54:40.262741 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 29628
I20260810 03:54:40.266343 30424 raft_consensus.cc:2258] T 4c0d3fe39fef446d8c2d0f88c403baaf P c8d6ed154f37483d9a7fe2e615a7d29d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:54:40.266871 30429 tablet_replica.cc:330] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e: stopping tablet replica
I20260810 03:54:40.267210 30429 raft_consensus.cc:2229] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:54:40.267652 30429 raft_consensus.cc:2258] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:54:40.270938 30429 ts_tablet_manager.cc:1905] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:54:40.270958 30424 ts_tablet_manager.cc:1905] T 4c0d3fe39fef446d8c2d0f88c403baaf P c8d6ed154f37483d9a7fe2e615a7d29d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:54:40.273849 30424 ts_tablet_manager.cc:1918] T 4c0d3fe39fef446d8c2d0f88c403baaf P c8d6ed154f37483d9a7fe2e615a7d29d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:54:40.273854 30429 ts_tablet_manager.cc:1918] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:54:40.273948 30424 log.cc:1196] T 4c0d3fe39fef446d8c2d0f88c403baaf P c8d6ed154f37483d9a7fe2e615a7d29d: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/4c0d3fe39fef446d8c2d0f88c403baaf
I20260810 03:54:40.273953 30429 log.cc:1196] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/4c0d3fe39fef446d8c2d0f88c403baaf
I20260810 03:54:40.274266 30429 ts_tablet_manager.cc:1939] T 4c0d3fe39fef446d8c2d0f88c403baaf P 1b941f423f0046e882a475818c14bf9e: Deleting consensus metadata
I20260810 03:54:40.274344 30424 ts_tablet_manager.cc:1939] T 4c0d3fe39fef446d8c2d0f88c403baaf P c8d6ed154f37483d9a7fe2e615a7d29d: Deleting consensus metadata
I20260810 03:54:40.275652 29494 catalog_manager.cc:4841] TS 1b941f423f0046e882a475818c14bf9e (127.27.153.66:45039): tablet 4c0d3fe39fef446d8c2d0f88c403baaf (table test-table [id=d33053f33838491caffd01d341ad5f48]) successfully deleted
I20260810 03:54:40.275892 29494 catalog_manager.cc:4841] TS c8d6ed154f37483d9a7fe2e615a7d29d (127.27.153.67:36971): tablet 4c0d3fe39fef446d8c2d0f88c403baaf (table test-table [id=d33053f33838491caffd01d341ad5f48]) successfully deleted
W20260810 03:54:40.276695 29496 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.153.65:43241 (error 108)
I20260810 03:54:40.277537 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 29882
I20260810 03:54:40.277751 30383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:40.277448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:43241 (local address 127.0.0.1:34046)
0810 03:54:40.277593 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:40.277666 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:43241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 03:54:40.299001 29496 catalog_manager.cc:4584] TS 18cd51281d524734b7488386bc3dbae1 (127.27.153.65:43241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c0d3fe39fef446d8c2d0f88c403baaf: Network error: Client connection negotiation failed: client connection to 127.27.153.65:43241: connect: Connection refused (error 111)
I20260810 03:54:40.304152 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 30134
I20260810 03:54:40.318884 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 29465
2026-08-10T03:54:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1171 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:54:40 dist-test-slave-pww0 krb5kdc[30437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:54:40 dist-test-slave-pww0 krb5kdc[30437](info): set up 2 sockets
Aug 10 03:54:40 dist-test-slave-pww0 krb5kdc[30437](info): commencing operation
krb5kdc: starting...
W20260810 03:54:41.527257 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:54:41 dist-test-slave-pww0 krb5kdc[30437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:54:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:54:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:54:41.658344 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:35243
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:35243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:54:41.774799 30457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:41.775115 30457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:41.775175 30457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:41.779286 30457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:54:41.779367 30457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:41.779395 30457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:41.779417 30457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:54:41.779438 30457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:54:41.784945 30457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:35243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:35243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:41.786100 30457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:41.787174 30457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:41.795959 30473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:41.795954 30468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:41.795954 30466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:41.796447 30457 server_base.cc:1034] running on GCE node
I20260810 03:54:41.797045 30457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:41.798264 30457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:41.799438 30457 hybrid_clock.cc:648] HybridClock initialized: now 1786334081799414 us; error 37 us; skew 500 ppm
Aug 10 03:54:41 dist-test-slave-pww0 krb5kdc[30437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334081, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:41.802287 30457 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:54:41.803282 30457 webserver.cc:460] Webserver started at http://127.27.153.126:34295/ using document root <none> and password file <none>
I20260810 03:54:41.803879 30457 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:41.803951 30457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:41.804163 30457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:41.806378 30457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "2228770a305f4aaaa1836b4f67f91e24"
format_stamp: "Formatted at 2026-08-10 03:54:41 on dist-test-slave-pww0"
server_key: "3136b904e4555116a9019f4db28b043e"
server_key_iv: "ae0cbb0493341ec026bd438021bf1d0c"
server_key_version: "encryptionkey@0"
I20260810 03:54:41.806949 30457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "2228770a305f4aaaa1836b4f67f91e24"
format_stamp: "Formatted at 2026-08-10 03:54:41 on dist-test-slave-pww0"
server_key: "3136b904e4555116a9019f4db28b043e"
server_key_iv: "ae0cbb0493341ec026bd438021bf1d0c"
server_key_version: "encryptionkey@0"
I20260810 03:54:41.810950 30457 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260810 03:54:41.813500 30482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:41.814750 30457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 03:54:41.814875 30457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "2228770a305f4aaaa1836b4f67f91e24"
format_stamp: "Formatted at 2026-08-10 03:54:41 on dist-test-slave-pww0"
server_key: "3136b904e4555116a9019f4db28b043e"
server_key_iv: "ae0cbb0493341ec026bd438021bf1d0c"
server_key_version: "encryptionkey@0"
I20260810 03:54:41.814978 30457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:41.831183 30457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:41.834573 30457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:41.834800 30457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:41.842202 30457 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:35243
I20260810 03:54:41.842238 30587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:35243 every 8 connection(s)
I20260810 03:54:41.843202 30457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:54:41.846004 30589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:41.846336 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 30457
I20260810 03:54:41.846459 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:54:41.846722 28261 external_mini_cluster.cc:1428] Setting key 1b1c932ece7f7b3c832bb56798a12e14
I20260810 03:54:41.852747 30589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24: Bootstrap starting.
I20260810 03:54:41.855252 30589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24: Neither blocks nor log segments found. Creating new log.
Aug 10 03:54:41 dist-test-slave-pww0 krb5kdc[30437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:54:41.856154 30589 log.cc:824] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:41.858343 30589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24: No bootstrap required, opened a new log
I20260810 03:54:41.860860 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:41.848143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.27.153.126:35243)
0810 03:54:41.848592 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:41.848608 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:41.848642 (+ 34us) server_negotiation.cc:407] Connection header received
0810 03:54:41.849334 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:41.849351 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:41.849701 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:41.850051 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:41.850917 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:41.851980 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:41.852830 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:41.853336 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:41.853486 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:41.856239 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:41.856247 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:41.856267 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:41.856306 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:41.857988 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:41.858435 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:41.858441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:41.858448 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:41.858583 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:41.858858 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:41.858861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:41.858863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:41.859276 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:41.859484 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:41.859790 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 03:54:41.860095 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":141,"threads_started":1}
I20260810 03:54:41.862453 30589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2228770a305f4aaaa1836b4f67f91e24" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 35243 } }
I20260810 03:54:41.862767 30589 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:41.862828 30589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2228770a305f4aaaa1836b4f67f91e24, State: Initialized, Role: FOLLOWER
I20260810 03:54:41.863303 30589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2228770a305f4aaaa1836b4f67f91e24" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 35243 } }
I20260810 03:54:41.863505 30589 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:54:41.863574 30589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:54:41.863651 30589 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:41.864626 30589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2228770a305f4aaaa1836b4f67f91e24" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 35243 } }
I20260810 03:54:41.864952 30589 leader_election.cc:302] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2228770a305f4aaaa1836b4f67f91e24; no voters:
I20260810 03:54:41.865238 30589 leader_election.cc:288] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:54:41.865375 30598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:54:41.865581 30598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24 [term 1 LEADER]: Becoming Leader. State: Replica: 2228770a305f4aaaa1836b4f67f91e24, State: Running, Role: LEADER
I20260810 03:54:41.865906 30598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2228770a305f4aaaa1836b4f67f91e24" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 35243 } }
I20260810 03:54:41.866475 30589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:54:41.869182 30603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2228770a305f4aaaa1836b4f67f91e24. Latest consensus state: current_term: 1 leader_uuid: "2228770a305f4aaaa1836b4f67f91e24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2228770a305f4aaaa1836b4f67f91e24" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 35243 } } }
I20260810 03:54:41.869177 30602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2228770a305f4aaaa1836b4f67f91e24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2228770a305f4aaaa1836b4f67f91e24" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 35243 } } }
I20260810 03:54:41.869463 30603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24 [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:41.869480 30602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24 [sys.catalog]: This master's current role is: LEADER
W20260810 03:54:41.872007 30625 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:54:41.872093 30625 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:54:41.872161 30627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:54:41.872862 30627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:54:41.878355 30627 catalog_manager.cc:1347] Generated new cluster ID: af0a5f913cfa4f98995cb43c52f5492f
I20260810 03:54:41.878430 30627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:54:41.898036 30627 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:54:41.899062 30627 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:54:41.906754 30627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24: Generated new TSK 0
I20260810 03:54:41.907689 30627 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.153.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
Entry for principal kudu/127.27.153.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab.
I20260810 03:54:41.942420 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:35243
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:54:42.059526 30634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:42.059819 30634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:42.059864 30634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:42.063958 30634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:42.064026 30634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:42.064106 30634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:54:42.069788 30634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.65.keytab
--principal=kudu/127.27.153.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:35243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:42.071273 30634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:42.072563 30634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:42.082942 30645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:42.082935 30644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:42.082957 30652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:42.083112 30634 server_base.cc:1034] running on GCE node
I20260810 03:54:42.083786 30634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:42.084439 30634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:42.085619 30634 hybrid_clock.cc:648] HybridClock initialized: now 1786334082085599 us; error 31 us; skew 500 ppm
Aug 10 03:54:42 dist-test-slave-pww0 krb5kdc[30437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334082, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:42.088855 30634 init.cc:377] Logged in from keytab as kudu/127.27.153.65@KRBTEST.COM (short username kudu)
I20260810 03:54:42.089905 30634 webserver.cc:460] Webserver started at http://127.27.153.65:41417/ using document root <none> and password file <none>
I20260810 03:54:42.090497 30634 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:42.090596 30634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:42.090819 30634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:42.093031 30634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "b36d2bc1362847af8f4eec431ecdfc5d"
format_stamp: "Formatted at 2026-08-10 03:54:42 on dist-test-slave-pww0"
server_key: "bce7646f48d3cf4e73c3404c6f038aa7"
server_key_iv: "cdad251788b9c5523313064e492a9cb6"
server_key_version: "encryptionkey@0"
I20260810 03:54:42.093622 30634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "b36d2bc1362847af8f4eec431ecdfc5d"
format_stamp: "Formatted at 2026-08-10 03:54:42 on dist-test-slave-pww0"
server_key: "bce7646f48d3cf4e73c3404c6f038aa7"
server_key_iv: "cdad251788b9c5523313064e492a9cb6"
server_key_version: "encryptionkey@0"
I20260810 03:54:42.097604 30634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:54:42.100489 30660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:42.101846 30634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:54:42.101975 30634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "b36d2bc1362847af8f4eec431ecdfc5d"
format_stamp: "Formatted at 2026-08-10 03:54:42 on dist-test-slave-pww0"
server_key: "bce7646f48d3cf4e73c3404c6f038aa7"
server_key_iv: "cdad251788b9c5523313064e492a9cb6"
server_key_version: "encryptionkey@0"
I20260810 03:54:42.102092 30634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:42.120675 30634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:42.124145 30634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:42.124358 30634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:42.125041 30634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:54:42.125941 30634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:54:42.125990 30634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:42.126046 30634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:54:42.126075 30634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:42.134871 30634 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:43441
I20260810 03:54:42.134902 30868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:43441 every 8 connection(s)
I20260810 03:54:42.135872 30634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:54:42.140178 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 30634
I20260810 03:54:42.140326 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:54:42.140672 28261 external_mini_cluster.cc:1428] Setting key 96cd4e4562f9e56459e96a664529a08d
Aug 10 03:54:42 dist-test-slave-pww0 krb5kdc[30437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334082, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.65@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:54:42.152269 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:42.137840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:49643 (local address 127.27.153.126:35243)
0810 03:54:42.137999 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:42.138004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:42.139013 (+ 1009us) server_negotiation.cc:407] Connection header received
0810 03:54:42.140549 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:42.140553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:42.140623 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:42.140729 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:42.142401 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.143134 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.144462 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.144755 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.144841 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:42.147748 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:42.147753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:42.147756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:42.147793 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:42.149528 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:42.150311 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:42.150315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:42.150317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:42.150430 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:42.150887 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:42.150891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:42.150893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:42.151158 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:42.151275 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:42.151914 (+ 639us) server_negotiation.cc:299] Negotiation successful
0810 03:54:42.152070 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.27.153.66@KRBTEST.COM; defaulting to no policy
I20260810 03:54:42.153088 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:42.138261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:35243 (local address 127.27.153.65:49643)
0810 03:54:42.138823 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:42.138875 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:42.140274 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:42.140939 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:42.140947 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:42.141428 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:54:42.142193 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:42.142219 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.143302 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.143305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:42.144317 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:42.144329 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.144922 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.144926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:42.145138 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:42.145818 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:54:42.145842 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:54:42.147513 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:54:42.149791 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:54:42.149797 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:54:42.149812 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:54:42.150158 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:54:42.150589 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:54:42.150592 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:54:42.150594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:54:42.150770 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:54:42.151257 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:54:42.151263 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:54:42.151622 (+ 359us) client_negotiation.cc:769] Sending connection context
0810 03:54:42.151815 (+ 193us) client_negotiation.cc:240] Negotiation successful
0810 03:54:42.152119 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":124,"threads_started":1}
I20260810 03:54:42.154421 30869 heartbeater.cc:347] Connected to a master server at 127.27.153.126:35243
I20260810 03:54:42.154724 30869 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.66@KRBTEST.COM" created.
I20260810 03:54:42.155290 30869 heartbeater.cc:511] Master 127.27.153.126:35243 requested a full tablet report, sending...
I20260810 03:54:42.157050 30505 ts_manager.cc:194] Registered new tserver with Master: b36d2bc1362847af8f4eec431ecdfc5d (127.27.153.65:43441)
I20260810 03:54:42.158632 30505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.65@KRBTEST.COM'} at 127.27.153.65:49643
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
Entry for principal kudu/127.27.153.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab.
I20260810 03:54:42.172912 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:35243
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:54:42.288729 30878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:42.289014 30878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:42.289059 30878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:42.293149 30878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:42.293220 30878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:42.293280 30878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:54:42.298715 30878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.66.keytab
--principal=kudu/127.27.153.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:35243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:42.299985 30878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:42.301023 30878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:42.311108 30886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:42.311136 30885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:42.311110 30890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:42.311380 30878 server_base.cc:1034] running on GCE node
I20260810 03:54:42.311856 30878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:42.312577 30878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:42.313771 30878 hybrid_clock.cc:648] HybridClock initialized: now 1786334082313726 us; error 34 us; skew 500 ppm
Aug 10 03:54:42 dist-test-slave-pww0 krb5kdc[30437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334082, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:42.316648 30878 init.cc:377] Logged in from keytab as kudu/127.27.153.66@KRBTEST.COM (short username kudu)
I20260810 03:54:42.317588 30878 webserver.cc:460] Webserver started at http://127.27.153.66:38749/ using document root <none> and password file <none>
I20260810 03:54:42.318167 30878 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:42.318249 30878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:42.318437 30878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:42.320773 30878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "a6e8c6b216694dcc900a0813b7c81c9c"
format_stamp: "Formatted at 2026-08-10 03:54:42 on dist-test-slave-pww0"
server_key: "977b9577e99b7e7ff03bf45b18457dfb"
server_key_iv: "677053c5bbd7d86f625b64725857a438"
server_key_version: "encryptionkey@0"
I20260810 03:54:42.321312 30878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "a6e8c6b216694dcc900a0813b7c81c9c"
format_stamp: "Formatted at 2026-08-10 03:54:42 on dist-test-slave-pww0"
server_key: "977b9577e99b7e7ff03bf45b18457dfb"
server_key_iv: "677053c5bbd7d86f625b64725857a438"
server_key_version: "encryptionkey@0"
I20260810 03:54:42.325202 30878 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:54:42.327945 30909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:42.329279 30878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:54:42.329396 30878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "a6e8c6b216694dcc900a0813b7c81c9c"
format_stamp: "Formatted at 2026-08-10 03:54:42 on dist-test-slave-pww0"
server_key: "977b9577e99b7e7ff03bf45b18457dfb"
server_key_iv: "677053c5bbd7d86f625b64725857a438"
server_key_version: "encryptionkey@0"
I20260810 03:54:42.329516 30878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:42.342175 30878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:42.345696 30878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:42.345890 30878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:42.346553 30878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:54:42.347450 30878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:54:42.347541 30878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:42.347609 30878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:54:42.347633 30878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:42.356787 30878 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:33447
I20260810 03:54:42.356823 31123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:33447 every 8 connection(s)
I20260810 03:54:42.357800 30878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:54:42.360839 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 30878
I20260810 03:54:42.360965 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:54:42.361220 28261 external_mini_cluster.cc:1428] Setting key bd51bf5dc3b15455da11de71326f57d1
Aug 10 03:54:42 dist-test-slave-pww0 krb5kdc[30437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334082, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.66@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.153.67@KRBTEST.COM; defaulting to no policy
I20260810 03:54:42.372586 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:42.359674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:40441 (local address 127.27.153.126:35243)
0810 03:54:42.359805 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:42.359810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:42.360786 (+ 976us) server_negotiation.cc:407] Connection header received
0810 03:54:42.362326 (+ 1540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:42.362329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:42.362375 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:42.362455 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:42.363730 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.364250 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.365424 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.365646 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.365738 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:42.368463 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:42.368467 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:42.368469 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:42.368496 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:42.370180 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:42.370743 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:42.370747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:42.370749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:42.370847 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:42.371268 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:42.371272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:42.371273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:42.371565 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:42.371675 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:42.372329 (+ 654us) server_negotiation.cc:299] Negotiation successful
0810 03:54:42.372458 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 03:54:42.373508 31127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:42.360056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:35243 (local address 127.27.153.66:40441)
0810 03:54:42.360620 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:42.360667 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:42.362087 (+ 1420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:42.362659 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:42.362668 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:42.363063 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:54:42.363580 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:42.363595 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.364381 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.364384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:42.365297 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:42.365308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.365802 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.365807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:42.365968 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:42.366620 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:54:42.366643 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:54:42.368280 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:54:42.370334 (+ 2054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:54:42.370339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:54:42.370353 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:54:42.370625 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:54:42.370965 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:54:42.370969 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:54:42.370970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:54:42.371146 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:54:42.371708 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:54:42.371715 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:54:42.372036 (+ 321us) client_negotiation.cc:769] Sending connection context
0810 03:54:42.372227 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 03:54:42.372550 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":183,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.67@KRBTEST.COM" created.
I20260810 03:54:42.374812 31125 heartbeater.cc:347] Connected to a master server at 127.27.153.126:35243
I20260810 03:54:42.375069 31125 heartbeater.cc:464] Registering TS with master...
I20260810 03:54:42.375648 31125 heartbeater.cc:511] Master 127.27.153.126:35243 requested a full tablet report, sending...
I20260810 03:54:42.376703 30505 ts_manager.cc:194] Registered new tserver with Master: a6e8c6b216694dcc900a0813b7c81c9c (127.27.153.66:33447)
I20260810 03:54:42.377342 30505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.66@KRBTEST.COM'} at 127.27.153.66:40441
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
Entry for principal kudu/127.27.153.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab.
I20260810 03:54:42.392306 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:35243
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:40785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:54:42.508361 31130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:42.508633 31130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:42.508679 31130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:42.512866 31130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:42.512944 31130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:42.513013 31130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:54:42.518729 31130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:40785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.67.keytab
--principal=kudu/127.27.153.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:35243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:42.520044 31130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:42.521152 31130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:42.531504 31143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:42.531504 31139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:42.531535 31146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:42.531795 31130 server_base.cc:1034] running on GCE node
I20260810 03:54:42.532349 31130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:42.533022 31130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:42.534188 31130 hybrid_clock.cc:648] HybridClock initialized: now 1786334082534167 us; error 30 us; skew 500 ppm
Aug 10 03:54:42 dist-test-slave-pww0 krb5kdc[30437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334082, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:42.537142 31130 init.cc:377] Logged in from keytab as kudu/127.27.153.67@KRBTEST.COM (short username kudu)
I20260810 03:54:42.538123 31130 webserver.cc:460] Webserver started at http://127.27.153.67:42971/ using document root <none> and password file <none>
I20260810 03:54:42.538697 31130 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:42.538770 31130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:42.538983 31130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:42.541201 31130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "d3ebd51caef843778bc6fa5ad93a4855"
format_stamp: "Formatted at 2026-08-10 03:54:42 on dist-test-slave-pww0"
server_key: "403af425b560ac0261d47933340d142a"
server_key_iv: "e40841bd23a33bd3f2d9adc7a0fc930a"
server_key_version: "encryptionkey@0"
I20260810 03:54:42.541797 31130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "d3ebd51caef843778bc6fa5ad93a4855"
format_stamp: "Formatted at 2026-08-10 03:54:42 on dist-test-slave-pww0"
server_key: "403af425b560ac0261d47933340d142a"
server_key_iv: "e40841bd23a33bd3f2d9adc7a0fc930a"
server_key_version: "encryptionkey@0"
I20260810 03:54:42.545780 31130 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:54:42.548475 31168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:42.549809 31130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 03:54:42.549947 31130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "d3ebd51caef843778bc6fa5ad93a4855"
format_stamp: "Formatted at 2026-08-10 03:54:42 on dist-test-slave-pww0"
server_key: "403af425b560ac0261d47933340d142a"
server_key_iv: "e40841bd23a33bd3f2d9adc7a0fc930a"
server_key_version: "encryptionkey@0"
I20260810 03:54:42.550063 31130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:42.575528 31130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:42.578784 31130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:42.578987 31130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:42.579656 31130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:54:42.580554 31130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:54:42.580602 31130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:42.580659 31130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:54:42.580689 31130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:42.589589 31130 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:34629
I20260810 03:54:42.589619 31372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:34629 every 8 connection(s)
I20260810 03:54:42.590548 31130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
Aug 10 03:54:42 dist-test-slave-pww0 krb5kdc[30437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334082, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.67@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:54:42.600409 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 31130
I20260810 03:54:42.600515 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:54:42.600847 28261 external_mini_cluster.cc:1428] Setting key 6a10de0f9f4a86284bfe53191e273e00
I20260810 03:54:42.604352 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:42.592434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:58637 (local address 127.27.153.126:35243)
0810 03:54:42.592619 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:42.592624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:42.593516 (+ 892us) server_negotiation.cc:407] Connection header received
0810 03:54:42.594817 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:42.594821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:42.594875 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:42.594953 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:42.596189 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.596731 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.597843 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.598072 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.598158 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:42.600739 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:42.600750 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:42.600753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:42.600787 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:42.602237 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:42.602856 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:42.602859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:42.602862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:42.602961 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:42.603297 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:42.603300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:42.603301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:42.603543 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:42.603619 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:42.604102 (+ 483us) server_negotiation.cc:299] Negotiation successful
0810 03:54:42.604210 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 03:54:42.605183 31377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:42.592881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:35243 (local address 127.27.153.67:58637)
0810 03:54:42.593370 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:42.593418 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:42.594615 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:42.595105 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:42.595111 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:42.595528 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:54:42.596048 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:42.596064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.596827 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.596830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:42.597722 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:42.597732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.598199 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.598201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:42.598361 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:42.598949 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:54:42.598966 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:54:42.600521 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:54:42.602350 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:54:42.602356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:54:42.602373 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:54:42.602749 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:54:42.603049 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:54:42.603053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:54:42.603054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:54:42.603211 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:54:42.603623 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:54:42.603628 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:54:42.603904 (+ 276us) client_negotiation.cc:769] Sending connection context
0810 03:54:42.604065 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 03:54:42.604342 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":117,"threads_started":1}
I20260810 03:54:42.606340 31373 heartbeater.cc:347] Connected to a master server at 127.27.153.126:35243
I20260810 03:54:42.606568 31373 heartbeater.cc:464] Registering TS with master...
I20260810 03:54:42.607084 31373 heartbeater.cc:511] Master 127.27.153.126:35243 requested a full tablet report, sending...
I20260810 03:54:42.608134 30505 ts_manager.cc:194] Registered new tserver with Master: d3ebd51caef843778bc6fa5ad93a4855 (127.27.153.67:34629)
I20260810 03:54:42.608680 30505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.153.67@KRBTEST.COM'} at 127.27.153.67:58637
I20260810 03:54:42.615016 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:54:42.624284 30596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:42.617053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56934 (local address 127.27.153.126:35243)
0810 03:54:42.617168 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:42.617172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:42.617348 (+ 176us) server_negotiation.cc:407] Connection header received
0810 03:54:42.617484 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:42.617487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:42.617536 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:42.617597 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:42.618359 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.618883 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.619827 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.620052 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.620140 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:42.621479 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:42.621483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:42.621486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:42.621514 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:42.622655 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:42.623048 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:42.623051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:42.623053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:42.623136 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:42.623386 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:42.623389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:42.623391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:42.623664 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:42.623794 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:42.624013 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 03:54:42.624128 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 03:54:42.627008 30505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:54:42.630085 30505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:54:42.645124 31405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:42.640272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.27.153.66:33447)
0810 03:54:42.640647 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:42.640658 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:42.640677 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:54:42.640760 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:42.640765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:42.640938 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:42.641329 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:42.641470 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.642502 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.643593 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.644360 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.644676 (+ 316us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:42.644816 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:42.644871 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 03:54:42.644967 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":145,"threads_started":1}
I20260810 03:54:42.645124 31400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:42.639516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58220 (local address 127.27.153.67:34629)
0810 03:54:42.640017 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:42.640030 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:42.640108 (+ 78us) server_negotiation.cc:407] Connection header received
0810 03:54:42.640333 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:42.640340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:42.640537 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:42.640676 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:42.641120 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.641882 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.643423 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.644312 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.644427 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:42.644591 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:42.644883 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 03:54:42.644971 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":162,"threads_started":1}
I20260810 03:54:42.645149 31404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:42.639847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:33447 (local address 127.0.0.1:33590)
0810 03:54:42.640152 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:42.640165 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:42.640252 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:42.641088 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:42.641091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:42.641108 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:42.641189 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:42.641194 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.642621 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.642624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:42.643433 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:42.643444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.644484 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.644487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:42.644571 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:42.644753 (+ 182us) client_negotiation.cc:769] Sending connection context
0810 03:54:42.644821 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 03:54:42.644976 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":134,"thread_start_us":100,"threads_started":1}
I20260810 03:54:42.645609 31397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:42.638957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58270 (local address 127.27.153.65:43441)
0810 03:54:42.640019 (+ 1062us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:42.640033 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:42.640058 (+ 25us) server_negotiation.cc:407] Connection header received
0810 03:54:42.640139 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:42.640145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:42.640396 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:42.640923 (+ 527us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:42.641078 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.642252 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.643671 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.644451 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.645189 (+ 738us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:42.645339 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:42.645388 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 03:54:42.645471 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":108,"threads_started":1}
I20260810 03:54:42.645674 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:42.639258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:34629 (local address 127.0.0.1:58220)
0810 03:54:42.640002 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:42.640015 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:42.640184 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:42.640855 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:42.640857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:42.640872 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:42.640994 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:42.641002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.642014 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.642016 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:42.643289 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:42.643299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.644438 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.644442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:42.644622 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:42.644701 (+ 79us) client_negotiation.cc:769] Sending connection context
0810 03:54:42.645346 (+ 645us) client_negotiation.cc:240] Negotiation successful
0810 03:54:42.645422 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"mutex_wait_us":59,"thread_start_us":114,"threads_started":1}
I20260810 03:54:42.645149 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:42.638825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:43441 (local address 127.0.0.1:58270)
0810 03:54:42.639378 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:42.639436 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:42.639600 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:42.640576 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:42.640582 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:42.640606 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:42.640772 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:42.640779 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.642405 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.642408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:42.643541 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:42.643551 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.644583 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.644586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:42.644751 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:42.644772 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 03:54:42.644821 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 03:54:42.644976 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":46,"thread_start_us":100,"threads_started":1}
I20260810 03:54:42.648627 31004 tablet_service.cc:1467] Processing CreateTablet for tablet f71fc582bba343bd87b5746790ded501 (DEFAULT_TABLE table=test-table [id=8f0836b4d8d149a58462c7a76cf66ca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:54:42.648705 31246 tablet_service.cc:1467] Processing CreateTablet for tablet f71fc582bba343bd87b5746790ded501 (DEFAULT_TABLE table=test-table [id=8f0836b4d8d149a58462c7a76cf66ca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:54:42.648705 30752 tablet_service.cc:1467] Processing CreateTablet for tablet f71fc582bba343bd87b5746790ded501 (DEFAULT_TABLE table=test-table [id=8f0836b4d8d149a58462c7a76cf66ca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:54:42.649662 31004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71fc582bba343bd87b5746790ded501. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:42.649760 31246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71fc582bba343bd87b5746790ded501. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:42.649791 30752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71fc582bba343bd87b5746790ded501. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:42.656961 31410 tablet_bootstrap.cc:492] T f71fc582bba343bd87b5746790ded501 P b36d2bc1362847af8f4eec431ecdfc5d: Bootstrap starting.
I20260810 03:54:42.657734 31411 tablet_bootstrap.cc:492] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855: Bootstrap starting.
I20260810 03:54:42.658174 31409 tablet_bootstrap.cc:492] T f71fc582bba343bd87b5746790ded501 P a6e8c6b216694dcc900a0813b7c81c9c: Bootstrap starting.
I20260810 03:54:42.659427 31410 tablet_bootstrap.cc:654] T f71fc582bba343bd87b5746790ded501 P b36d2bc1362847af8f4eec431ecdfc5d: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:42.659560 31411 tablet_bootstrap.cc:654] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:42.660293 31409 tablet_bootstrap.cc:654] T f71fc582bba343bd87b5746790ded501 P a6e8c6b216694dcc900a0813b7c81c9c: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:42.660357 31410 log.cc:824] T f71fc582bba343bd87b5746790ded501 P b36d2bc1362847af8f4eec431ecdfc5d: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:42.660631 31411 log.cc:824] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:42.661492 31409 log.cc:824] T f71fc582bba343bd87b5746790ded501 P a6e8c6b216694dcc900a0813b7c81c9c: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:42.662552 31410 tablet_bootstrap.cc:492] T f71fc582bba343bd87b5746790ded501 P b36d2bc1362847af8f4eec431ecdfc5d: No bootstrap required, opened a new log
I20260810 03:54:42.662762 31410 ts_tablet_manager.cc:1397] T f71fc582bba343bd87b5746790ded501 P b36d2bc1362847af8f4eec431ecdfc5d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:54:42.663077 31411 tablet_bootstrap.cc:492] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855: No bootstrap required, opened a new log
I20260810 03:54:42.663262 31411 ts_tablet_manager.cc:1397] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260810 03:54:42.663420 31409 tablet_bootstrap.cc:492] T f71fc582bba343bd87b5746790ded501 P a6e8c6b216694dcc900a0813b7c81c9c: No bootstrap required, opened a new log
I20260810 03:54:42.663632 31409 ts_tablet_manager.cc:1397] T f71fc582bba343bd87b5746790ded501 P a6e8c6b216694dcc900a0813b7c81c9c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:54:42.666159 31411 raft_consensus.cc:348] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36d2bc1362847af8f4eec431ecdfc5d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43441 } } peers { permanent_uuid: "a6e8c6b216694dcc900a0813b7c81c9c" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 33447 } } peers { permanent_uuid: "d3ebd51caef843778bc6fa5ad93a4855" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 34629 } }
I20260810 03:54:42.666430 31411 raft_consensus.cc:374] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:42.666484 31411 raft_consensus.cc:729] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3ebd51caef843778bc6fa5ad93a4855, State: Initialized, Role: FOLLOWER
I20260810 03:54:42.666930 31411 consensus_queue.cc:260] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36d2bc1362847af8f4eec431ecdfc5d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43441 } } peers { permanent_uuid: "a6e8c6b216694dcc900a0813b7c81c9c" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 33447 } } peers { permanent_uuid: "d3ebd51caef843778bc6fa5ad93a4855" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 34629 } }
I20260810 03:54:42.667471 31410 raft_consensus.cc:348] T f71fc582bba343bd87b5746790ded501 P b36d2bc1362847af8f4eec431ecdfc5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36d2bc1362847af8f4eec431ecdfc5d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43441 } } peers { permanent_uuid: "a6e8c6b216694dcc900a0813b7c81c9c" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 33447 } } peers { permanent_uuid: "d3ebd51caef843778bc6fa5ad93a4855" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 34629 } }
I20260810 03:54:42.667474 31409 raft_consensus.cc:348] T f71fc582bba343bd87b5746790ded501 P a6e8c6b216694dcc900a0813b7c81c9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36d2bc1362847af8f4eec431ecdfc5d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43441 } } peers { permanent_uuid: "a6e8c6b216694dcc900a0813b7c81c9c" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 33447 } } peers { permanent_uuid: "d3ebd51caef843778bc6fa5ad93a4855" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 34629 } }
I20260810 03:54:42.667719 31373 heartbeater.cc:503] Master 127.27.153.126:35243 was elected leader, sending a full tablet report...
I20260810 03:54:42.667740 31410 raft_consensus.cc:374] T f71fc582bba343bd87b5746790ded501 P b36d2bc1362847af8f4eec431ecdfc5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:42.667750 31409 raft_consensus.cc:374] T f71fc582bba343bd87b5746790ded501 P a6e8c6b216694dcc900a0813b7c81c9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:42.667793 31410 raft_consensus.cc:729] T f71fc582bba343bd87b5746790ded501 P b36d2bc1362847af8f4eec431ecdfc5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36d2bc1362847af8f4eec431ecdfc5d, State: Initialized, Role: FOLLOWER
I20260810 03:54:42.667793 31409 raft_consensus.cc:729] T f71fc582bba343bd87b5746790ded501 P a6e8c6b216694dcc900a0813b7c81c9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e8c6b216694dcc900a0813b7c81c9c, State: Initialized, Role: FOLLOWER
I20260810 03:54:42.668016 31411 ts_tablet_manager.cc:1428] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260810 03:54:42.668273 31410 consensus_queue.cc:260] T f71fc582bba343bd87b5746790ded501 P b36d2bc1362847af8f4eec431ecdfc5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36d2bc1362847af8f4eec431ecdfc5d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43441 } } peers { permanent_uuid: "a6e8c6b216694dcc900a0813b7c81c9c" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 33447 } } peers { permanent_uuid: "d3ebd51caef843778bc6fa5ad93a4855" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 34629 } }
I20260810 03:54:42.668272 31409 consensus_queue.cc:260] T f71fc582bba343bd87b5746790ded501 P a6e8c6b216694dcc900a0813b7c81c9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36d2bc1362847af8f4eec431ecdfc5d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43441 } } peers { permanent_uuid: "a6e8c6b216694dcc900a0813b7c81c9c" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 33447 } } peers { permanent_uuid: "d3ebd51caef843778bc6fa5ad93a4855" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 34629 } }
I20260810 03:54:42.669293 31409 ts_tablet_manager.cc:1428] T f71fc582bba343bd87b5746790ded501 P a6e8c6b216694dcc900a0813b7c81c9c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 03:54:42.669528 30869 heartbeater.cc:503] Master 127.27.153.126:35243 was elected leader, sending a full tablet report...
I20260810 03:54:42.669618 31125 heartbeater.cc:503] Master 127.27.153.126:35243 was elected leader, sending a full tablet report...
I20260810 03:54:42.671039 31410 ts_tablet_manager.cc:1428] T f71fc582bba343bd87b5746790ded501 P b36d2bc1362847af8f4eec431ecdfc5d: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260810 03:54:42.727304 31416 raft_consensus.cc:482] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:54:42.727591 31416 raft_consensus.cc:504] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36d2bc1362847af8f4eec431ecdfc5d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43441 } } peers { permanent_uuid: "a6e8c6b216694dcc900a0813b7c81c9c" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 33447 } } peers { permanent_uuid: "d3ebd51caef843778bc6fa5ad93a4855" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 34629 } }
I20260810 03:54:42.728789 31416 leader_election.cc:288] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b36d2bc1362847af8f4eec431ecdfc5d (127.27.153.65:43441), a6e8c6b216694dcc900a0813b7c81c9c (127.27.153.66:33447)
I20260810 03:54:42.732491 31377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:42.729110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:43441 (local address 127.27.153.67:47955)
0810 03:54:42.729284 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:42.729302 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:42.729427 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:42.729756 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:42.729759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:42.729779 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:42.729887 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:42.729892 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.730641 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.730644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:42.731432 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:42.731442 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.732084 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.732087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:42.732190 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:42.732208 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 03:54:42.732271 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 03:54:42.732348 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 03:54:42.732491 31397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:42.729257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:47955 (local address 127.27.153.65:43441)
0810 03:54:42.729407 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:42.729412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:42.729431 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:54:42.729543 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:42.729546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:42.729638 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:42.729725 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:42.729997 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.730523 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.731567 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.731969 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.732080 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:42.732136 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:42.732295 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 03:54:42.732369 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 03:54:42.732933 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:42.729403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:33447 (local address 127.27.153.67:43013)
0810 03:54:42.729791 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:42.729801 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:42.729900 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:42.730122 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:42.730125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:42.730141 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:42.730260 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:42.730266 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.730964 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.730968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:42.731792 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:42.731801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.732400 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.732404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:42.732498 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:42.732516 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 03:54:42.732725 (+ 209us) client_negotiation.cc:240] Negotiation successful
0810 03:54:42.732796 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":140,"threads_started":1}
I20260810 03:54:42.732961 31405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:42.729539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:43013 (local address 127.27.153.66:33447)
0810 03:54:42.729701 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:42.729706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:42.729883 (+ 177us) server_negotiation.cc:407] Connection header received
0810 03:54:42.729987 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:42.729990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:42.730033 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:42.730097 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:42.730384 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.730868 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.731904 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.732298 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.732401 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:42.732452 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:42.732767 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 03:54:42.732849 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 03:54:42.733232 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71fc582bba343bd87b5746790ded501" candidate_uuid: "d3ebd51caef843778bc6fa5ad93a4855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36d2bc1362847af8f4eec431ecdfc5d" is_pre_election: true
I20260810 03:54:42.733515 30793 raft_consensus.cc:2454] T f71fc582bba343bd87b5746790ded501 P b36d2bc1362847af8f4eec431ecdfc5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3ebd51caef843778bc6fa5ad93a4855 in term 0.
I20260810 03:54:42.733985 31169 leader_election.cc:302] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b36d2bc1362847af8f4eec431ecdfc5d, d3ebd51caef843778bc6fa5ad93a4855; no voters:
I20260810 03:54:42.734293 31416 raft_consensus.cc:2781] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:54:42.734380 31416 raft_consensus.cc:482] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:54:42.734412 31416 raft_consensus.cc:3037] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:42.734359 31045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71fc582bba343bd87b5746790ded501" candidate_uuid: "d3ebd51caef843778bc6fa5ad93a4855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e8c6b216694dcc900a0813b7c81c9c" is_pre_election: true
I20260810 03:54:42.734652 31045 raft_consensus.cc:2454] T f71fc582bba343bd87b5746790ded501 P a6e8c6b216694dcc900a0813b7c81c9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3ebd51caef843778bc6fa5ad93a4855 in term 0.
I20260810 03:54:42.735656 31416 raft_consensus.cc:504] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36d2bc1362847af8f4eec431ecdfc5d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43441 } } peers { permanent_uuid: "a6e8c6b216694dcc900a0813b7c81c9c" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 33447 } } peers { permanent_uuid: "d3ebd51caef843778bc6fa5ad93a4855" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 34629 } }
I20260810 03:54:42.735999 31416 leader_election.cc:288] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855 [CANDIDATE]: Term 1 election: Requested vote from peers b36d2bc1362847af8f4eec431ecdfc5d (127.27.153.65:43441), a6e8c6b216694dcc900a0813b7c81c9c (127.27.153.66:33447)
I20260810 03:54:42.736405 30793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71fc582bba343bd87b5746790ded501" candidate_uuid: "d3ebd51caef843778bc6fa5ad93a4855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36d2bc1362847af8f4eec431ecdfc5d"
I20260810 03:54:42.736424 31045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71fc582bba343bd87b5746790ded501" candidate_uuid: "d3ebd51caef843778bc6fa5ad93a4855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e8c6b216694dcc900a0813b7c81c9c"
I20260810 03:54:42.736552 31045 raft_consensus.cc:3037] T f71fc582bba343bd87b5746790ded501 P a6e8c6b216694dcc900a0813b7c81c9c [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:42.736552 30793 raft_consensus.cc:3037] T f71fc582bba343bd87b5746790ded501 P b36d2bc1362847af8f4eec431ecdfc5d [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:42.737908 31045 raft_consensus.cc:2454] T f71fc582bba343bd87b5746790ded501 P a6e8c6b216694dcc900a0813b7c81c9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ebd51caef843778bc6fa5ad93a4855 in term 1.
I20260810 03:54:42.737908 30793 raft_consensus.cc:2454] T f71fc582bba343bd87b5746790ded501 P b36d2bc1362847af8f4eec431ecdfc5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3ebd51caef843778bc6fa5ad93a4855 in term 1.
I20260810 03:54:42.738222 31169 leader_election.cc:302] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b36d2bc1362847af8f4eec431ecdfc5d, d3ebd51caef843778bc6fa5ad93a4855; no voters:
I20260810 03:54:42.738442 31416 raft_consensus.cc:2781] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:54:42.738747 31416 raft_consensus.cc:686] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855 [term 1 LEADER]: Becoming Leader. State: Replica: d3ebd51caef843778bc6fa5ad93a4855, State: Running, Role: LEADER
I20260810 03:54:42.739130 31416 consensus_queue.cc:237] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36d2bc1362847af8f4eec431ecdfc5d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43441 } } peers { permanent_uuid: "a6e8c6b216694dcc900a0813b7c81c9c" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 33447 } } peers { permanent_uuid: "d3ebd51caef843778bc6fa5ad93a4855" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 34629 } }
I20260810 03:54:42.742491 30498 catalog_manager.cc:5495] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3ebd51caef843778bc6fa5ad93a4855 (127.27.153.67). New cstate: current_term: 1 leader_uuid: "d3ebd51caef843778bc6fa5ad93a4855" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36d2bc1362847af8f4eec431ecdfc5d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e8c6b216694dcc900a0813b7c81c9c" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 33447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3ebd51caef843778bc6fa5ad93a4855" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 34629 } health_report { overall_health: HEALTHY } } }
I20260810 03:54:42.770257 31400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:42.766507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58228 (local address 127.27.153.67:34629)
0810 03:54:42.766636 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:42.766640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:42.766654 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:54:42.766806 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:42.766809 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:42.766869 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:42.766966 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:54:42.767220 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.767863 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.768774 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:42.768989 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:42.769080 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:42.769396 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:54:42.769832 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:54:42.769943 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:42.770015 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 03:54:42.770081 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 03:54:42.776517 31045 raft_consensus.cc:1261] T f71fc582bba343bd87b5746790ded501 P a6e8c6b216694dcc900a0813b7c81c9c [term 1 FOLLOWER]: Refusing update from remote peer d3ebd51caef843778bc6fa5ad93a4855: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:54:42.776517 30793 raft_consensus.cc:1261] T f71fc582bba343bd87b5746790ded501 P b36d2bc1362847af8f4eec431ecdfc5d [term 1 FOLLOWER]: Refusing update from remote peer d3ebd51caef843778bc6fa5ad93a4855: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:54:42.777299 31416 consensus_queue.cc:1035] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b36d2bc1362847af8f4eec431ecdfc5d" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 43441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:54:42.777572 31421 consensus_queue.cc:1035] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e8c6b216694dcc900a0813b7c81c9c" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 33447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:54:42.785504 31426 mvcc.cc:205] Tried to move back new op lower bound from 7316824403044896768 to 7316824402902650880. Current Snapshot: MvccSnapshot[applied={T|T < 7316824403044896768}]
I20260810 03:54:42.787093 31429 mvcc.cc:205] Tried to move back new op lower bound from 7316824403044896768 to 7316824402902650880. Current Snapshot: MvccSnapshot[applied={T|T < 7316824403044896768 or (T in {7316824403044896768})}]
I20260810 03:54:42.795490 30498 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:54:42.795655 30498 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:54:42.798271 30498 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2228770a305f4aaaa1836b4f67f91e24: Sending DeleteTablet for 3 replicas of tablet f71fc582bba343bd87b5746790ded501
I20260810 03:54:42.799078 31246 tablet_service.cc:1514] Processing DeleteTablet for tablet f71fc582bba343bd87b5746790ded501 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:54:42 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:58220
I20260810 03:54:42.799294 30752 tablet_service.cc:1514] Processing DeleteTablet for tablet f71fc582bba343bd87b5746790ded501 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:54:42 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:58270
I20260810 03:54:42.799407 31004 tablet_service.cc:1514] Processing DeleteTablet for tablet f71fc582bba343bd87b5746790ded501 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:54:42 UTC) from {username='kudu', principal='kudu/127.27.153.126@KRBTEST.COM'} at 127.0.0.1:33590
I20260810 03:54:42.800153 31446 tablet_replica.cc:330] T f71fc582bba343bd87b5746790ded501 P a6e8c6b216694dcc900a0813b7c81c9c: stopping tablet replica
I20260810 03:54:42.800269 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 30634
I20260810 03:54:42.800516 31446 raft_consensus.cc:2229] T f71fc582bba343bd87b5746790ded501 P a6e8c6b216694dcc900a0813b7c81c9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:54:42.808024 31446 raft_consensus.cc:2258] T f71fc582bba343bd87b5746790ded501 P a6e8c6b216694dcc900a0813b7c81c9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:54:42.813663 31446 ts_tablet_manager.cc:1905] T f71fc582bba343bd87b5746790ded501 P a6e8c6b216694dcc900a0813b7c81c9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:54:42.817099 31446 ts_tablet_manager.cc:1918] T f71fc582bba343bd87b5746790ded501 P a6e8c6b216694dcc900a0813b7c81c9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:54:42.817260 31446 log.cc:1196] T f71fc582bba343bd87b5746790ded501 P a6e8c6b216694dcc900a0813b7c81c9c: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/f71fc582bba343bd87b5746790ded501
I20260810 03:54:42.817665 31446 ts_tablet_manager.cc:1939] T f71fc582bba343bd87b5746790ded501 P a6e8c6b216694dcc900a0813b7c81c9c: Deleting consensus metadata
I20260810 03:54:42.818132 31444 tablet_replica.cc:330] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855: stopping tablet replica
I20260810 03:54:42.818638 31444 raft_consensus.cc:2229] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:54:42.819140 31444 raft_consensus.cc:2258] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:54:42.820866 30483 catalog_manager.cc:4841] TS a6e8c6b216694dcc900a0813b7c81c9c (127.27.153.66:33447): tablet f71fc582bba343bd87b5746790ded501 (table test-table [id=8f0836b4d8d149a58462c7a76cf66ca0]) successfully deleted
I20260810 03:54:42.821460 31444 ts_tablet_manager.cc:1905] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:54:42.824334 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 30878
W20260810 03:54:42.824306 30483 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.153.65:43441 (error 108)
I20260810 03:54:42.839322 31444 ts_tablet_manager.cc:1918] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:54:42.839540 31444 log.cc:1196] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/f71fc582bba343bd87b5746790ded501
I20260810 03:54:42.839898 31444 ts_tablet_manager.cc:1939] T f71fc582bba343bd87b5746790ded501 P d3ebd51caef843778bc6fa5ad93a4855: Deleting consensus metadata
I20260810 03:54:42.841241 30487 catalog_manager.cc:4841] TS d3ebd51caef843778bc6fa5ad93a4855 (127.27.153.67:34629): tablet f71fc582bba343bd87b5746790ded501 (table test-table [id=8f0836b4d8d149a58462c7a76cf66ca0]) successfully deleted
I20260810 03:54:42.842486 31396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:42.842121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:43441 (local address 127.0.0.1:58282)
0810 03:54:42.842322 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:42.842400 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:43441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260810 03:54:42.842653 30483 catalog_manager.cc:4584] TS b36d2bc1362847af8f4eec431ecdfc5d (127.27.153.65:43441): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f71fc582bba343bd87b5746790ded501: Network error: Client connection negotiation failed: client connection to 127.27.153.65:43441: connect: Connection refused (error 111)
I20260810 03:54:42.865607 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 31130
I20260810 03:54:42.877069 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 30457
2026-08-10T03:54:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2563 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T03:54:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:54:42Z Disabled control of system clock
I20260810 03:54:42.925477 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:43423
--webserver_interface=127.27.153.126
--webserver_port=0
--builtin_ntp_servers=127.27.153.84:42345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:43423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:54:43.040397 31455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:43.040735 31455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:43.040784 31455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:43.044891 31455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:54:43.044971 31455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:43.044986 31455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:43.045001 31455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:54:43.045015 31455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:54:43.050031 31455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:42345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:43423
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:43423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:43.051267 31455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:43.052724 31455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:43.061501 31464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:43.061510 31470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:43.061518 31463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:43.061861 31455 server_base.cc:1034] running on GCE node
I20260810 03:54:43.062456 31455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:43.063695 31455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:43.064893 31455 hybrid_clock.cc:648] HybridClock initialized: now 1786334083064874 us; error 33 us; skew 500 ppm
I20260810 03:54:43.073735 31455 webserver.cc:460] Webserver started at http://127.27.153.126:42047/ using document root <none> and password file <none>
I20260810 03:54:43.074415 31455 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:43.074497 31455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:43.074723 31455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:43.077128 31455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "e36ce845a9ea4d20b8dd81e1cf94eb63"
format_stamp: "Formatted at 2026-08-10 03:54:43 on dist-test-slave-pww0"
server_key: "7d0b7da97e49529a8e35cdcf13b181d1"
server_key_iv: "5164533b0b202d65a666c96f06a25997"
server_key_version: "encryptionkey@0"
I20260810 03:54:43.077778 31455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "e36ce845a9ea4d20b8dd81e1cf94eb63"
format_stamp: "Formatted at 2026-08-10 03:54:43 on dist-test-slave-pww0"
server_key: "7d0b7da97e49529a8e35cdcf13b181d1"
server_key_iv: "5164533b0b202d65a666c96f06a25997"
server_key_version: "encryptionkey@0"
I20260810 03:54:43.082070 31455 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:54:43.084805 31484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:43.086139 31455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:54:43.086289 31455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "e36ce845a9ea4d20b8dd81e1cf94eb63"
format_stamp: "Formatted at 2026-08-10 03:54:43 on dist-test-slave-pww0"
server_key: "7d0b7da97e49529a8e35cdcf13b181d1"
server_key_iv: "5164533b0b202d65a666c96f06a25997"
server_key_version: "encryptionkey@0"
I20260810 03:54:43.086404 31455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:43.101682 31455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:43.102501 31455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:43.102690 31455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:43.109961 31455 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:43423
I20260810 03:54:43.109985 31575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:43423 every 8 connection(s)
I20260810 03:54:43.110967 31455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:54:43.112042 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 31455
I20260810 03:54:43.112155 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:54:43.112442 28261 external_mini_cluster.cc:1428] Setting key 57215783546378b0a41fe7e5399babfb
I20260810 03:54:43.114526 31578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:43.121668 31578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63: Bootstrap starting.
I20260810 03:54:43.123070 31582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:43.113880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56308 (local address 127.27.153.126:43423)
0810 03:54:43.114605 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:43.114625 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:43.114670 (+ 45us) server_negotiation.cc:407] Connection header received
0810 03:54:43.115560 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:43.115584 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:43.116089 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:43.116476 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:43.116915 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.118393 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.119321 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.120037 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.120207 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:43.120999 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:43.121005 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:43.121022 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:43.121529 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:43.121598 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:43.121625 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:43.121726 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:43.121845 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 03:54:43.122123 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":114,"threads_started":1}
I20260810 03:54:43.124317 31578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:43.125275 31578 log.cc:824] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:43.127831 31578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63: No bootstrap required, opened a new log
I20260810 03:54:43.132390 31578 raft_consensus.cc:348] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36ce845a9ea4d20b8dd81e1cf94eb63" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 43423 } }
I20260810 03:54:43.132706 31578 raft_consensus.cc:374] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:43.132788 31578 raft_consensus.cc:729] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e36ce845a9ea4d20b8dd81e1cf94eb63, State: Initialized, Role: FOLLOWER
I20260810 03:54:43.133280 31578 consensus_queue.cc:260] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36ce845a9ea4d20b8dd81e1cf94eb63" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 43423 } }
I20260810 03:54:43.133430 31578 raft_consensus.cc:388] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:54:43.133494 31578 raft_consensus.cc:482] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:54:43.133594 31578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:43.134788 31578 raft_consensus.cc:504] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36ce845a9ea4d20b8dd81e1cf94eb63" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 43423 } }
I20260810 03:54:43.135128 31578 leader_election.cc:302] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e36ce845a9ea4d20b8dd81e1cf94eb63; no voters:
I20260810 03:54:43.135433 31578 leader_election.cc:288] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:54:43.135600 31584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:54:43.135818 31584 raft_consensus.cc:686] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63 [term 1 LEADER]: Becoming Leader. State: Replica: e36ce845a9ea4d20b8dd81e1cf94eb63, State: Running, Role: LEADER
I20260810 03:54:43.136122 31584 consensus_queue.cc:237] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36ce845a9ea4d20b8dd81e1cf94eb63" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 43423 } }
I20260810 03:54:43.136696 31578 sys_catalog.cc:564] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:54:43.137712 31586 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e36ce845a9ea4d20b8dd81e1cf94eb63. Latest consensus state: current_term: 1 leader_uuid: "e36ce845a9ea4d20b8dd81e1cf94eb63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36ce845a9ea4d20b8dd81e1cf94eb63" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 43423 } } }
I20260810 03:54:43.137843 31586 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63 [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:43.137974 31585 sys_catalog.cc:455] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e36ce845a9ea4d20b8dd81e1cf94eb63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e36ce845a9ea4d20b8dd81e1cf94eb63" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 43423 } } }
I20260810 03:54:43.138108 31585 sys_catalog.cc:458] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63 [sys.catalog]: This master's current role is: LEADER
W20260810 03:54:43.141837 31619 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:54:43.141918 31619 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:54:43.141996 31606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:54:43.142712 31606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:54:43.148561 31606 catalog_manager.cc:1347] Generated new cluster ID: 5a9c40f487e94567a9244069470e6420
I20260810 03:54:43.148635 31606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:54:43.161152 31606 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:54:43.162168 31606 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:54:43.169821 31606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63: Generated new TSK 0
I20260810 03:54:43.170612 31606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:54:43.173699 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.65:0
--local_ip_for_outbound_sockets=127.27.153.65
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:43423
--builtin_ntp_servers=127.27.153.84:42345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 03:54:43.315161 31629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:43.315486 31629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:43.315546 31629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:43.319736 31629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:43.319818 31629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:43.319877 31629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.65
I20260810 03:54:43.325064 31629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:42345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.153.65
--webserver_port=0
--tserver_master_addrs=127.27.153.126:43423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.65
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:43.326448 31629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:43.327947 31629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:43.337877 31635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:43.337870 31639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:43.337965 31636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:43.338152 31629 server_base.cc:1034] running on GCE node
I20260810 03:54:43.338634 31629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:43.339389 31629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:43.340556 31629 hybrid_clock.cc:648] HybridClock initialized: now 1786334083340536 us; error 33 us; skew 500 ppm
I20260810 03:54:43.342491 31629 webserver.cc:460] Webserver started at http://127.27.153.65:37331/ using document root <none> and password file <none>
I20260810 03:54:43.343120 31629 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:43.343184 31629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:43.343393 31629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:43.345608 31629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-0/data/instance:
uuid: "a6572a7401804fe8aa10f86a740ba978"
format_stamp: "Formatted at 2026-08-10 03:54:43 on dist-test-slave-pww0"
server_key: "1dd7411a29d28ebbf6f527636bd3de6d"
server_key_iv: "77df76c8db6b1eb296cefd1567877bef"
server_key_version: "encryptionkey@0"
I20260810 03:54:43.346189 31629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance:
uuid: "a6572a7401804fe8aa10f86a740ba978"
format_stamp: "Formatted at 2026-08-10 03:54:43 on dist-test-slave-pww0"
server_key: "1dd7411a29d28ebbf6f527636bd3de6d"
server_key_iv: "77df76c8db6b1eb296cefd1567877bef"
server_key_version: "encryptionkey@0"
I20260810 03:54:43.350097 31629 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:54:43.352794 31649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:43.354139 31629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:54:43.354314 31629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-0/wal
uuid: "a6572a7401804fe8aa10f86a740ba978"
format_stamp: "Formatted at 2026-08-10 03:54:43 on dist-test-slave-pww0"
server_key: "1dd7411a29d28ebbf6f527636bd3de6d"
server_key_iv: "77df76c8db6b1eb296cefd1567877bef"
server_key_version: "encryptionkey@0"
I20260810 03:54:43.354452 31629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:43.374249 31629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:43.374989 31629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:43.375155 31629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:43.375834 31629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:54:43.376884 31629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:54:43.376932 31629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:43.376973 31629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:54:43.376993 31629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:43.387805 31629 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.65:36183
I20260810 03:54:43.387880 31851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.65:36183 every 8 connection(s)
I20260810 03:54:43.388789 31629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-0/data/info.pb
I20260810 03:54:43.392584 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 31629
I20260810 03:54:43.392691 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-0/wal/instance
I20260810 03:54:43.392972 28261 external_mini_cluster.cc:1428] Setting key 37fd6b3003f8a491dcdf0d4941f9f447
I20260810 03:54:43.395001 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.66:0
--local_ip_for_outbound_sockets=127.27.153.66
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:43423
--builtin_ntp_servers=127.27.153.84:42345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:54:43.398527 31582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:43.390738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.65:41179 (local address 127.27.153.126:43423)
0810 03:54:43.390895 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:43.390900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:43.392096 (+ 1196us) server_negotiation.cc:407] Connection header received
0810 03:54:43.393527 (+ 1431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:43.393530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:43.393588 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:43.393683 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:43.394808 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.395399 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.396588 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.396824 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.396916 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:43.398006 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:43.398010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:43.398013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:43.398028 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:43.398052 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:43.398061 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:43.398126 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:43.398290 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 03:54:43.398377 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 03:54:43.399387 31855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:43.391212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:43423 (local address 127.27.153.65:41179)
0810 03:54:43.391945 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:43.391989 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:43.393289 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:43.393897 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:43.393906 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:43.394018 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:54:43.394654 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:43.394669 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.395529 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.395532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:43.396464 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:43.396474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.396978 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.396982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:43.397140 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:43.397737 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:54:43.397738 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:54:43.397784 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:54:43.397852 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:54:43.398143 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:54:43.398153 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:54:43.398158 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 03:54:43.398284 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 03:54:43.398535 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":157,"threads_started":1}
I20260810 03:54:43.400740 31852 heartbeater.cc:347] Connected to a master server at 127.27.153.126:43423
I20260810 03:54:43.401018 31852 heartbeater.cc:464] Registering TS with master...
I20260810 03:54:43.401722 31852 heartbeater.cc:511] Master 127.27.153.126:43423 requested a full tablet report, sending...
I20260810 03:54:43.403707 31511 ts_manager.cc:194] Registered new tserver with Master: a6572a7401804fe8aa10f86a740ba978 (127.27.153.65:36183)
I20260810 03:54:43.405421 31511 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.153.65:41179
W20260810 03:54:43.514647 31856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:43.514979 31856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:43.515023 31856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:43.519217 31856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:43.519284 31856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:43.519340 31856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.66
I20260810 03:54:43.524255 31856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:42345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.153.66
--webserver_port=0
--tserver_master_addrs=127.27.153.126:43423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.66
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:43.525389 31856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:43.526816 31856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:54:43.537093 31856 server_base.cc:1034] running on GCE node
W20260810 03:54:43.537024 31866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:43.537024 31871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:43.537024 31867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:43.537853 31856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:43.538568 31856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:43.539775 31856 hybrid_clock.cc:648] HybridClock initialized: now 1786334083539761 us; error 45 us; skew 500 ppm
I20260810 03:54:43.541854 31856 webserver.cc:460] Webserver started at http://127.27.153.66:45867/ using document root <none> and password file <none>
I20260810 03:54:43.542505 31856 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:43.542578 31856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:43.542819 31856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:43.545028 31856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-1/data/instance:
uuid: "2e3c6e7e9d0c4d5d8b2a29e829d127a3"
format_stamp: "Formatted at 2026-08-10 03:54:43 on dist-test-slave-pww0"
server_key: "1d56e1bc7bd7427788606d151bb57d77"
server_key_iv: "eaaec0517d9f4be9b6d4828bd4728d9a"
server_key_version: "encryptionkey@0"
I20260810 03:54:43.545604 31856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance:
uuid: "2e3c6e7e9d0c4d5d8b2a29e829d127a3"
format_stamp: "Formatted at 2026-08-10 03:54:43 on dist-test-slave-pww0"
server_key: "1d56e1bc7bd7427788606d151bb57d77"
server_key_iv: "eaaec0517d9f4be9b6d4828bd4728d9a"
server_key_version: "encryptionkey@0"
I20260810 03:54:43.549448 31856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:54:43.552114 31879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:43.553488 31856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:54:43.553613 31856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-1/wal
uuid: "2e3c6e7e9d0c4d5d8b2a29e829d127a3"
format_stamp: "Formatted at 2026-08-10 03:54:43 on dist-test-slave-pww0"
server_key: "1d56e1bc7bd7427788606d151bb57d77"
server_key_iv: "eaaec0517d9f4be9b6d4828bd4728d9a"
server_key_version: "encryptionkey@0"
I20260810 03:54:43.553735 31856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:43.568296 31856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:43.569069 31856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:43.569267 31856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:43.569916 31856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:54:43.570981 31856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:54:43.571033 31856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:43.571089 31856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:54:43.571117 31856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:43.579873 31856 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.66:46855
I20260810 03:54:43.579905 32072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.66:46855 every 8 connection(s)
I20260810 03:54:43.580863 31856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-1/data/info.pb
I20260810 03:54:43.583246 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 31856
I20260810 03:54:43.583371 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-1/wal/instance
I20260810 03:54:43.583688 28261 external_mini_cluster.cc:1428] Setting key 377ccb9651fd685da24a473f319f575d
I20260810 03:54:43.585716 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.153.67:0
--local_ip_for_outbound_sockets=127.27.153.67
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:43423
--builtin_ntp_servers=127.27.153.84:42345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 03:54:43.591302 31582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:43.582737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:49453 (local address 127.27.153.126:43423)
0810 03:54:43.582874 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:43.582878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:43.584122 (+ 1244us) server_negotiation.cc:407] Connection header received
0810 03:54:43.585564 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:43.585568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:43.585626 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:43.585712 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:43.586705 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.587411 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.589090 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.589323 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.589405 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:43.590805 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:43.590809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:43.590811 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:43.590825 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:43.590845 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:43.590853 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:43.590918 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:43.591108 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 03:54:43.591181 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 03:54:43.592593 32076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:43.583178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:43423 (local address 127.27.153.66:49453)
0810 03:54:43.583942 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:43.583997 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:43.585345 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:43.585922 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:43.585930 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:43.586025 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:54:43.586560 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:43.586574 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.587590 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.587594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:43.588952 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:43.588964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.589484 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.589488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:43.589718 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:43.590408 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:54:43.590410 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:54:43.590476 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:54:43.590584 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:54:43.590944 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:54:43.590954 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:54:43.590961 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 03:54:43.591104 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 03:54:43.591442 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":136,"threads_started":1}
I20260810 03:54:43.594121 32073 heartbeater.cc:347] Connected to a master server at 127.27.153.126:43423
I20260810 03:54:43.594390 32073 heartbeater.cc:464] Registering TS with master...
I20260810 03:54:43.594977 32073 heartbeater.cc:511] Master 127.27.153.126:43423 requested a full tablet report, sending...
I20260810 03:54:43.596149 31511 ts_manager.cc:194] Registered new tserver with Master: 2e3c6e7e9d0c4d5d8b2a29e829d127a3 (127.27.153.66:46855)
I20260810 03:54:43.596735 31511 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.153.66:49453
W20260810 03:54:43.705798 32077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:43.706103 32077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:43.706161 32077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:43.710291 32077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:43.710374 32077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:43.710453 32077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.153.67
I20260810 03:54:43.715405 32077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:42345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.153.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.153.67
--webserver_port=0
--tserver_master_addrs=127.27.153.126:43423
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.153.67
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:43.716572 32077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:43.718001 32077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:43.727764 32087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:43.727799 32089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:43.727993 32077 server_base.cc:1034] running on GCE node
W20260810 03:54:43.727823 32086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:43.728511 32077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:43.729192 32077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:43.730415 32077 hybrid_clock.cc:648] HybridClock initialized: now 1786334083730398 us; error 28 us; skew 500 ppm
I20260810 03:54:43.732486 32077 webserver.cc:460] Webserver started at http://127.27.153.67:37705/ using document root <none> and password file <none>
I20260810 03:54:43.733208 32077 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:43.733305 32077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:43.733520 32077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:43.735731 32077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-2/data/instance:
uuid: "c24f165fa33c4661bbd7657b55f7dc42"
format_stamp: "Formatted at 2026-08-10 03:54:43 on dist-test-slave-pww0"
server_key: "066c06a062d6f8f54160350f46b36c9c"
server_key_iv: "0eff776aa1ab011f18145ab974dab464"
server_key_version: "encryptionkey@0"
I20260810 03:54:43.736322 32077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance:
uuid: "c24f165fa33c4661bbd7657b55f7dc42"
format_stamp: "Formatted at 2026-08-10 03:54:43 on dist-test-slave-pww0"
server_key: "066c06a062d6f8f54160350f46b36c9c"
server_key_iv: "0eff776aa1ab011f18145ab974dab464"
server_key_version: "encryptionkey@0"
I20260810 03:54:43.740299 32077 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:54:43.742823 32097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:43.744098 32077 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:54:43.744212 32077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-2/wal
uuid: "c24f165fa33c4661bbd7657b55f7dc42"
format_stamp: "Formatted at 2026-08-10 03:54:43 on dist-test-slave-pww0"
server_key: "066c06a062d6f8f54160350f46b36c9c"
server_key_iv: "0eff776aa1ab011f18145ab974dab464"
server_key_version: "encryptionkey@0"
I20260810 03:54:43.744321 32077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:43.783010 32077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:43.783854 32077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:43.784049 32077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:43.784739 32077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 03:54:43.785763 32077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 03:54:43.785813 32077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:43.785859 32077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 03:54:43.785872 32077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:43.795141 32077 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.67:41215
I20260810 03:54:43.795169 32289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.67:41215 every 8 connection(s)
I20260810 03:54:43.796200 32077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-2/data/info.pb
I20260810 03:54:43.804764 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 32077
I20260810 03:54:43.804872 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-2/wal/instance
I20260810 03:54:43.805152 28261 external_mini_cluster.cc:1428] Setting key 2c462c8a48fcd2df6b4a1f256c9946b6
I20260810 03:54:43.806026 31582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:43.798203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.67:55203 (local address 127.27.153.126:43423)
0810 03:54:43.798341 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:43.798345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:43.799510 (+ 1165us) server_negotiation.cc:407] Connection header received
0810 03:54:43.800966 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:43.800970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:43.801031 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:43.801108 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:43.802174 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.802707 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.803950 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.804170 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.804264 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:43.805439 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:43.805443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:43.805445 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:43.805462 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:43.805487 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:43.805498 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:43.805568 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:43.805812 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 03:54:43.805883 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 03:54:43.807128 32295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:43.798600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:43423 (local address 127.27.153.67:55203)
0810 03:54:43.799325 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:43.799370 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:43.800736 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:43.801316 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:43.801324 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:43.801419 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:54:43.802020 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:43.802035 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.802839 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.802842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:43.803822 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:43.803834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.804328 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.804332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:43.804492 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:43.805133 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 03:54:43.805134 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:54:43.805189 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 03:54:43.805282 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:54:43.805595 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:54:43.805631 (+ 36us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:54:43.805639 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 03:54:43.805805 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 03:54:43.806112 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":134,"threads_started":1}
I20260810 03:54:43.808333 32293 heartbeater.cc:347] Connected to a master server at 127.27.153.126:43423
I20260810 03:54:43.808585 32293 heartbeater.cc:464] Registering TS with master...
I20260810 03:54:43.809150 32293 heartbeater.cc:511] Master 127.27.153.126:43423 requested a full tablet report, sending...
I20260810 03:54:43.810264 31511 ts_manager.cc:194] Registered new tserver with Master: c24f165fa33c4661bbd7657b55f7dc42 (127.27.153.67:41215)
I20260810 03:54:43.810853 31511 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.153.67:55203
I20260810 03:54:43.819845 28261 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 03:54:43.825673 31582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:43.821896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56324 (local address 127.27.153.126:43423)
0810 03:54:43.822015 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:43.822019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:43.822171 (+ 152us) server_negotiation.cc:407] Connection header received
0810 03:54:43.822327 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:43.822331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:43.822379 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:43.822440 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:43.822738 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.823266 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.824171 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.824396 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.824485 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:43.825275 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:43.825279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:43.825281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:43.825296 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:43.825317 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:43.825325 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:43.825391 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:43.825472 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 03:54:43.825531 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 03:54:43.828662 31511 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 03:54:43.831761 31511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 03:54:43.846393 32314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:43.840699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49124 (local address 127.27.153.66:46855)
0810 03:54:43.841171 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:43.841183 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:43.841210 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:54:43.841445 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:43.841451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:43.841670 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:43.841840 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:43.842342 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.843319 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.844954 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.845729 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.845826 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:43.846037 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:43.846131 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 03:54:43.846230 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":163,"threads_started":1}
I20260810 03:54:43.846390 32312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:43.840561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.66:46855 (local address 127.0.0.1:49124)
0810 03:54:43.841046 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:43.841096 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:43.841227 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:43.842035 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:43.842041 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:43.842067 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:43.842205 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:43.842212 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.843420 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.843424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:43.844825 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:43.844836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.845841 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.845845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:43.845947 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:43.845986 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 03:54:43.846099 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 03:54:43.846236 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":24,"thread_start_us":86,"threads_started":1}
I20260810 03:54:43.846750 32316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:43.840674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:41215 (local address 127.0.0.1:50926)
0810 03:54:43.842129 (+ 1455us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:43.842140 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:43.842210 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:43.842666 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:43.842669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:43.842685 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:43.842772 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:43.842778 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.844154 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.844157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:43.845196 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:43.845203 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.846356 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.846360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:43.846446 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:43.846470 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 03:54:43.846535 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 03:54:43.846606 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1400,"mutex_wait_us":30,"thread_start_us":128,"threads_started":1}
I20260810 03:54:43.847306 32320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:43.841736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50926 (local address 127.27.153.67:41215)
0810 03:54:43.842182 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:43.842193 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:43.842218 (+ 25us) server_negotiation.cc:407] Connection header received
0810 03:54:43.842351 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:43.842357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:43.842532 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:43.842683 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:43.842882 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.844023 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.845313 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.846004 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.846106 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:43.846287 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:43.847037 (+ 750us) server_negotiation.cc:299] Negotiation successful
0810 03:54:43.847144 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":147,"threads_started":1}
I20260810 03:54:43.847662 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:43.841534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56630 (local address 127.27.153.65:36183)
0810 03:54:43.842002 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:43.842014 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:43.842495 (+ 481us) server_negotiation.cc:407] Connection header received
0810 03:54:43.842573 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:43.842577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:43.842753 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:43.842894 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:43.843294 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.844089 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.845561 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.846326 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.846421 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:43.846968 (+ 547us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:43.847271 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 03:54:43.847376 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":149,"threads_started":1}
I20260810 03:54:43.849808 31962 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe10c7aba834bb8974dfca1dc9e1fa6 (DEFAULT_TABLE table=test-table [id=e26054d32ad14fbf89987b427892e262]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:54:43.850127 32182 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe10c7aba834bb8974dfca1dc9e1fa6 (DEFAULT_TABLE table=test-table [id=e26054d32ad14fbf89987b427892e262]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:54:43.851073 31962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe10c7aba834bb8974dfca1dc9e1fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:43.851073 32182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe10c7aba834bb8974dfca1dc9e1fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:43.851819 32310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:43.841310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:36183 (local address 127.0.0.1:56630)
0810 03:54:43.842384 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:43.842397 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:43.842485 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:43.843044 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:43.843047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:43.843064 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:43.843159 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:43.843165 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.844232 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.844236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:43.845430 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:43.845441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.847032 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.847036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:43.847116 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:43.847138 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 03:54:43.847213 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 03:54:43.847288 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"mutex_wait_us":52,"thread_start_us":100,"threads_started":1}
I20260810 03:54:43.854391 31736 tablet_service.cc:1467] Processing CreateTablet for tablet 3fe10c7aba834bb8974dfca1dc9e1fa6 (DEFAULT_TABLE table=test-table [id=e26054d32ad14fbf89987b427892e262]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 03:54:43.855372 31736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fe10c7aba834bb8974dfca1dc9e1fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:43.859341 32322 tablet_bootstrap.cc:492] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3: Bootstrap starting.
I20260810 03:54:43.859637 32321 tablet_bootstrap.cc:492] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P c24f165fa33c4661bbd7657b55f7dc42: Bootstrap starting.
I20260810 03:54:43.862167 32322 tablet_bootstrap.cc:654] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:43.863117 32322 log.cc:824] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:43.864825 32321 tablet_bootstrap.cc:654] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P c24f165fa33c4661bbd7657b55f7dc42: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:43.865494 32322 tablet_bootstrap.cc:492] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3: No bootstrap required, opened a new log
I20260810 03:54:43.865718 32322 ts_tablet_manager.cc:1397] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20260810 03:54:43.865751 32321 log.cc:824] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P c24f165fa33c4661bbd7657b55f7dc42: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:43.866564 32323 tablet_bootstrap.cc:492] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P a6572a7401804fe8aa10f86a740ba978: Bootstrap starting.
I20260810 03:54:43.867863 32321 tablet_bootstrap.cc:492] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P c24f165fa33c4661bbd7657b55f7dc42: No bootstrap required, opened a new log
I20260810 03:54:43.868069 32321 ts_tablet_manager.cc:1397] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P c24f165fa33c4661bbd7657b55f7dc42: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260810 03:54:43.869127 32323 tablet_bootstrap.cc:654] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P a6572a7401804fe8aa10f86a740ba978: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:43.870076 32323 log.cc:824] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P a6572a7401804fe8aa10f86a740ba978: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:43.870232 32322 raft_consensus.cc:348] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c6e7e9d0c4d5d8b2a29e829d127a3" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 46855 } } peers { permanent_uuid: "c24f165fa33c4661bbd7657b55f7dc42" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41215 } } peers { permanent_uuid: "a6572a7401804fe8aa10f86a740ba978" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 36183 } }
I20260810 03:54:43.870546 32322 raft_consensus.cc:374] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:43.870600 32322 raft_consensus.cc:729] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e3c6e7e9d0c4d5d8b2a29e829d127a3, State: Initialized, Role: FOLLOWER
I20260810 03:54:43.871122 32322 consensus_queue.cc:260] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c6e7e9d0c4d5d8b2a29e829d127a3" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 46855 } } peers { permanent_uuid: "c24f165fa33c4661bbd7657b55f7dc42" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41215 } } peers { permanent_uuid: "a6572a7401804fe8aa10f86a740ba978" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 36183 } }
I20260810 03:54:43.871852 32073 heartbeater.cc:503] Master 127.27.153.126:43423 was elected leader, sending a full tablet report...
I20260810 03:54:43.872232 32322 ts_tablet_manager.cc:1428] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3: Time spent starting tablet: real 0.006s user 0.003s sys 0.005s
I20260810 03:54:43.872072 32321 raft_consensus.cc:348] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P c24f165fa33c4661bbd7657b55f7dc42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c6e7e9d0c4d5d8b2a29e829d127a3" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 46855 } } peers { permanent_uuid: "c24f165fa33c4661bbd7657b55f7dc42" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41215 } } peers { permanent_uuid: "a6572a7401804fe8aa10f86a740ba978" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 36183 } }
I20260810 03:54:43.872344 32321 raft_consensus.cc:374] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P c24f165fa33c4661bbd7657b55f7dc42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:43.872395 32321 raft_consensus.cc:729] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P c24f165fa33c4661bbd7657b55f7dc42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c24f165fa33c4661bbd7657b55f7dc42, State: Initialized, Role: FOLLOWER
I20260810 03:54:43.872668 32323 tablet_bootstrap.cc:492] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P a6572a7401804fe8aa10f86a740ba978: No bootstrap required, opened a new log
I20260810 03:54:43.872876 32323 ts_tablet_manager.cc:1397] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P a6572a7401804fe8aa10f86a740ba978: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 03:54:43.872850 32321 consensus_queue.cc:260] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P c24f165fa33c4661bbd7657b55f7dc42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c6e7e9d0c4d5d8b2a29e829d127a3" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 46855 } } peers { permanent_uuid: "c24f165fa33c4661bbd7657b55f7dc42" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41215 } } peers { permanent_uuid: "a6572a7401804fe8aa10f86a740ba978" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 36183 } }
I20260810 03:54:43.873546 32293 heartbeater.cc:503] Master 127.27.153.126:43423 was elected leader, sending a full tablet report...
I20260810 03:54:43.873929 32321 ts_tablet_manager.cc:1428] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P c24f165fa33c4661bbd7657b55f7dc42: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 03:54:43.877097 32323 raft_consensus.cc:348] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P a6572a7401804fe8aa10f86a740ba978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c6e7e9d0c4d5d8b2a29e829d127a3" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 46855 } } peers { permanent_uuid: "c24f165fa33c4661bbd7657b55f7dc42" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41215 } } peers { permanent_uuid: "a6572a7401804fe8aa10f86a740ba978" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 36183 } }
I20260810 03:54:43.877416 32323 raft_consensus.cc:374] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P a6572a7401804fe8aa10f86a740ba978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:43.877487 32323 raft_consensus.cc:729] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P a6572a7401804fe8aa10f86a740ba978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6572a7401804fe8aa10f86a740ba978, State: Initialized, Role: FOLLOWER
I20260810 03:54:43.877939 32323 consensus_queue.cc:260] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P a6572a7401804fe8aa10f86a740ba978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c6e7e9d0c4d5d8b2a29e829d127a3" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 46855 } } peers { permanent_uuid: "c24f165fa33c4661bbd7657b55f7dc42" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41215 } } peers { permanent_uuid: "a6572a7401804fe8aa10f86a740ba978" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 36183 } }
I20260810 03:54:43.878710 31852 heartbeater.cc:503] Master 127.27.153.126:43423 was elected leader, sending a full tablet report...
I20260810 03:54:43.879012 32323 ts_tablet_manager.cc:1428] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P a6572a7401804fe8aa10f86a740ba978: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260810 03:54:43.890003 31854 tablet.cc:2369] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P a6572a7401804fe8aa10f86a740ba978: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:54:43.913388 32330 raft_consensus.cc:482] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 03:54:43.913600 32330 raft_consensus.cc:504] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c6e7e9d0c4d5d8b2a29e829d127a3" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 46855 } } peers { permanent_uuid: "c24f165fa33c4661bbd7657b55f7dc42" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41215 } } peers { permanent_uuid: "a6572a7401804fe8aa10f86a740ba978" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 36183 } }
I20260810 03:54:43.914736 32330 leader_election.cc:288] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c24f165fa33c4661bbd7657b55f7dc42 (127.27.153.67:41215), a6572a7401804fe8aa10f86a740ba978 (127.27.153.65:36183)
I20260810 03:54:43.918345 32076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:43.915019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.67:41215 (local address 127.27.153.66:32999)
0810 03:54:43.915171 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:43.915189 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:43.915320 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:43.915657 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:43.915660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:43.915679 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:43.915786 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:43.915792 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.916511 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.916515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:43.917307 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:43.917317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.917961 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.917965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:43.918060 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:43.918087 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 03:54:43.918162 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 03:54:43.918233 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":1}
I20260810 03:54:43.918383 32320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:43.915143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:32999 (local address 127.27.153.67:41215)
0810 03:54:43.915295 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:43.915300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:43.915320 (+ 20us) server_negotiation.cc:407] Connection header received
0810 03:54:43.915425 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:43.915428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:43.915506 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:43.915589 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:43.915894 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.916409 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.917433 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.917834 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.917946 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:43.917997 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:43.918182 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 03:54:43.918253 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:54:43.919020 32333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:43.915328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:36183 (local address 127.27.153.66:49069)
0810 03:54:43.915708 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:43.915719 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 03:54:43.915813 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:54:43.916028 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:54:43.916030 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:54:43.916045 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 03:54:43.916156 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:43.916161 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.916874 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.916878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:43.917683 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:54:43.917694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.918602 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.918605 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:54:43.918691 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:43.918713 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 03:54:43.918819 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 03:54:43.918884 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":157,"threads_started":1}
I20260810 03:54:43.919044 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:43.915514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.153.66:49069 (local address 127.27.153.65:36183)
0810 03:54:43.915648 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:43.915652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:43.915788 (+ 136us) server_negotiation.cc:407] Connection header received
0810 03:54:43.915896 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:43.915899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:43.915945 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:43.916011 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 03:54:43.916254 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.916774 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.917810 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.918355 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.918469 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:43.918513 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:43.918851 (+ 338us) server_negotiation.cc:299] Negotiation successful
0810 03:54:43.918933 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 03:54:43.919083 32214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe10c7aba834bb8974dfca1dc9e1fa6" candidate_uuid: "2e3c6e7e9d0c4d5d8b2a29e829d127a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c24f165fa33c4661bbd7657b55f7dc42" is_pre_election: true
I20260810 03:54:43.919391 32214 raft_consensus.cc:2454] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P c24f165fa33c4661bbd7657b55f7dc42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e3c6e7e9d0c4d5d8b2a29e829d127a3 in term 0.
I20260810 03:54:43.919849 31776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe10c7aba834bb8974dfca1dc9e1fa6" candidate_uuid: "2e3c6e7e9d0c4d5d8b2a29e829d127a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6572a7401804fe8aa10f86a740ba978" is_pre_election: true
I20260810 03:54:43.919899 31882 leader_election.cc:302] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e3c6e7e9d0c4d5d8b2a29e829d127a3, c24f165fa33c4661bbd7657b55f7dc42; no voters:
I20260810 03:54:43.920140 31776 raft_consensus.cc:2454] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P a6572a7401804fe8aa10f86a740ba978 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e3c6e7e9d0c4d5d8b2a29e829d127a3 in term 0.
I20260810 03:54:43.920172 32330 raft_consensus.cc:2781] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 03:54:43.920269 32330 raft_consensus.cc:482] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 03:54:43.920341 32330 raft_consensus.cc:3037] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:43.921386 32330 raft_consensus.cc:504] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c6e7e9d0c4d5d8b2a29e829d127a3" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 46855 } } peers { permanent_uuid: "c24f165fa33c4661bbd7657b55f7dc42" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41215 } } peers { permanent_uuid: "a6572a7401804fe8aa10f86a740ba978" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 36183 } }
I20260810 03:54:43.921720 32330 leader_election.cc:288] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3 [CANDIDATE]: Term 1 election: Requested vote from peers c24f165fa33c4661bbd7657b55f7dc42 (127.27.153.67:41215), a6572a7401804fe8aa10f86a740ba978 (127.27.153.65:36183)
I20260810 03:54:43.922044 32214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe10c7aba834bb8974dfca1dc9e1fa6" candidate_uuid: "2e3c6e7e9d0c4d5d8b2a29e829d127a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c24f165fa33c4661bbd7657b55f7dc42"
I20260810 03:54:43.922139 31776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fe10c7aba834bb8974dfca1dc9e1fa6" candidate_uuid: "2e3c6e7e9d0c4d5d8b2a29e829d127a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6572a7401804fe8aa10f86a740ba978"
I20260810 03:54:43.922176 32214 raft_consensus.cc:3037] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P c24f165fa33c4661bbd7657b55f7dc42 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:43.922253 31776 raft_consensus.cc:3037] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P a6572a7401804fe8aa10f86a740ba978 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:43.923489 31776 raft_consensus.cc:2454] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P a6572a7401804fe8aa10f86a740ba978 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e3c6e7e9d0c4d5d8b2a29e829d127a3 in term 1.
I20260810 03:54:43.923489 32214 raft_consensus.cc:2454] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P c24f165fa33c4661bbd7657b55f7dc42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e3c6e7e9d0c4d5d8b2a29e829d127a3 in term 1.
I20260810 03:54:43.923816 31882 leader_election.cc:302] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e3c6e7e9d0c4d5d8b2a29e829d127a3, a6572a7401804fe8aa10f86a740ba978; no voters:
I20260810 03:54:43.923990 32330 raft_consensus.cc:2781] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:54:43.924250 32330 raft_consensus.cc:686] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3 [term 1 LEADER]: Becoming Leader. State: Replica: 2e3c6e7e9d0c4d5d8b2a29e829d127a3, State: Running, Role: LEADER
I20260810 03:54:43.924602 32330 consensus_queue.cc:237] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c6e7e9d0c4d5d8b2a29e829d127a3" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 46855 } } peers { permanent_uuid: "c24f165fa33c4661bbd7657b55f7dc42" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41215 } } peers { permanent_uuid: "a6572a7401804fe8aa10f86a740ba978" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 36183 } }
I20260810 03:54:43.927743 31511 catalog_manager.cc:5495] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e3c6e7e9d0c4d5d8b2a29e829d127a3 (127.27.153.66). New cstate: current_term: 1 leader_uuid: "2e3c6e7e9d0c4d5d8b2a29e829d127a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e3c6e7e9d0c4d5d8b2a29e829d127a3" member_type: VOTER last_known_addr { host: "127.27.153.66" port: 46855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c24f165fa33c4661bbd7657b55f7dc42" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6572a7401804fe8aa10f86a740ba978" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 36183 } health_report { overall_health: UNKNOWN } } }
I20260810 03:54:43.946314 32314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:43.943161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49136 (local address 127.27.153.66:46855)
0810 03:54:43.943292 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:43.943296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:43.943309 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:54:43.943389 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:43.943391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:43.943447 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:43.943547 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:54:43.943759 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.944259 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.945053 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:43.945264 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:43.945329 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:43.945567 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:54:43.946002 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:54:43.946091 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:43.946121 (+ 30us) server_negotiation.cc:299] Negotiation successful
0810 03:54:43.946171 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:54:43.952682 32214 raft_consensus.cc:1261] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P c24f165fa33c4661bbd7657b55f7dc42 [term 1 FOLLOWER]: Refusing update from remote peer 2e3c6e7e9d0c4d5d8b2a29e829d127a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:54:43.952682 31776 raft_consensus.cc:1261] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P a6572a7401804fe8aa10f86a740ba978 [term 1 FOLLOWER]: Refusing update from remote peer 2e3c6e7e9d0c4d5d8b2a29e829d127a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:54:43.953475 32334 consensus_queue.cc:1035] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24f165fa33c4661bbd7657b55f7dc42" member_type: VOTER last_known_addr { host: "127.27.153.67" port: 41215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:54:43.953675 32330 consensus_queue.cc:1035] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6572a7401804fe8aa10f86a740ba978" member_type: VOTER last_known_addr { host: "127.27.153.65" port: 36183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:54:43.962404 32342 mvcc.cc:205] Tried to move back new op lower bound from 7316824407862022144 to 7316824407757799424. Current Snapshot: MvccSnapshot[applied={T|T < 7316824407862022144}]
I20260810 03:54:43.962376 32343 mvcc.cc:205] Tried to move back new op lower bound from 7316824407862022144 to 7316824407757799424. Current Snapshot: MvccSnapshot[applied={T|T < 7316824407862022144}]
I20260810 03:54:43.972565 31508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:54:43.972755 31508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 03:54:43.975346 31508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e36ce845a9ea4d20b8dd81e1cf94eb63: Sending DeleteTablet for 3 replicas of tablet 3fe10c7aba834bb8974dfca1dc9e1fa6
I20260810 03:54:43.976432 32182 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fe10c7aba834bb8974dfca1dc9e1fa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:54:43 UTC) from {username='slave'} at 127.0.0.1:50926
I20260810 03:54:43.976562 31962 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fe10c7aba834bb8974dfca1dc9e1fa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:54:43 UTC) from {username='slave'} at 127.0.0.1:49124
I20260810 03:54:43.976567 31736 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fe10c7aba834bb8974dfca1dc9e1fa6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 03:54:43 UTC) from {username='slave'} at 127.0.0.1:56630
I20260810 03:54:43.977120 32363 tablet_replica.cc:330] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P c24f165fa33c4661bbd7657b55f7dc42: stopping tablet replica
I20260810 03:54:43.977224 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 31629
I20260810 03:54:43.977499 32363 raft_consensus.cc:2229] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P c24f165fa33c4661bbd7657b55f7dc42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 03:54:43.977864 32363 raft_consensus.cc:2258] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P c24f165fa33c4661bbd7657b55f7dc42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:54:43.980654 32365 tablet_replica.cc:330] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3: stopping tablet replica
I20260810 03:54:43.981118 32365 raft_consensus.cc:2229] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 03:54:43.981683 32365 raft_consensus.cc:2258] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 03:54:43.986331 32365 ts_tablet_manager.cc:1905] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:54:43.989807 32365 ts_tablet_manager.cc:1918] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:54:43.989956 32365 log.cc:1196] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-1/wal/wals/3fe10c7aba834bb8974dfca1dc9e1fa6
I20260810 03:54:43.990407 32365 ts_tablet_manager.cc:1939] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P 2e3c6e7e9d0c4d5d8b2a29e829d127a3: Deleting consensus metadata
I20260810 03:54:43.992112 31488 catalog_manager.cc:4841] TS 2e3c6e7e9d0c4d5d8b2a29e829d127a3 (127.27.153.66:46855): tablet 3fe10c7aba834bb8974dfca1dc9e1fa6 (table test-table [id=e26054d32ad14fbf89987b427892e262]) successfully deleted
I20260810 03:54:43.993287 32363 ts_tablet_manager.cc:1905] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P c24f165fa33c4661bbd7657b55f7dc42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 03:54:43.996634 32363 ts_tablet_manager.cc:1918] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P c24f165fa33c4661bbd7657b55f7dc42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 03:54:43.996784 32363 log.cc:1196] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P c24f165fa33c4661bbd7657b55f7dc42: Deleting WAL directory at /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786333814837068-28261-0/minicluster-data/ts-2/wal/wals/3fe10c7aba834bb8974dfca1dc9e1fa6
I20260810 03:54:43.997150 32363 ts_tablet_manager.cc:1939] T 3fe10c7aba834bb8974dfca1dc9e1fa6 P c24f165fa33c4661bbd7657b55f7dc42: Deleting consensus metadata
I20260810 03:54:44.001749 31485 catalog_manager.cc:4841] TS c24f165fa33c4661bbd7657b55f7dc42 (127.27.153.67:41215): tablet 3fe10c7aba834bb8974dfca1dc9e1fa6 (table test-table [id=e26054d32ad14fbf89987b427892e262]) successfully deleted
I20260810 03:54:44.009526 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 31856
W20260810 03:54:44.009505 31485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.153.65:36183 (error 108)
I20260810 03:54:44.037263 32310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:44.035662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.65:36183 (local address 127.0.0.1:56644)
0810 03:54:44.037049 (+ 1387us) negotiation.cc:107] Waiting for socket to connect
0810 03:54:44.037161 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.65:36183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1283}
W20260810 03:54:44.038558 31485 catalog_manager.cc:4584] TS a6572a7401804fe8aa10f86a740ba978 (127.27.153.65:36183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3fe10c7aba834bb8974dfca1dc9e1fa6: Network error: Client connection negotiation failed: client connection to 127.27.153.65:36183: connect: Connection refused (error 111)
I20260810 03:54:44.043694 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 32077
I20260810 03:54:44.062912 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 31455
2026-08-10T03:54:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1175 ms)
[----------] 6 tests from EncryptionPolicyTest (11430 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:54:44 dist-test-slave-pww0 krb5kdc[32369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:54:44 dist-test-slave-pww0 krb5kdc[32369](info): set up 2 sockets
Aug 10 03:54:44 dist-test-slave-pww0 krb5kdc[32369](info): commencing operation
krb5kdc: starting...
W20260810 03:54:45.267908 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.176s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:54:45 dist-test-slave-pww0 krb5kdc[32369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:54:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:54:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:54:45.400043 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33441
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:54:45.516412 32393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:45.516716 32393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:45.516780 32393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:45.520947 32393 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:54:45.521025 32393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:45.521054 32393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:45.521076 32393 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:54:45.521106 32393 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:54:45.526646 32393 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33441
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:45.527833 32393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:45.528892 32393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:45.537753 32403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:45.537781 32408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:45.537887 32393 server_base.cc:1034] running on GCE node
W20260810 03:54:45.537757 32404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:45.539439 32393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:45.540671 32393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:45.541868 32393 hybrid_clock.cc:648] HybridClock initialized: now 1786334085541839 us; error 42 us; skew 500 ppm
Aug 10 03:54:45 dist-test-slave-pww0 krb5kdc[32369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:45.544780 32393 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:54:45.545739 32393 webserver.cc:460] Webserver started at http://127.0.0.1:37245/ using document root <none> and password file <none>
I20260810 03:54:45.546283 32393 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:45.546350 32393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:45.546523 32393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:45.548636 32393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "b4e1108fbd6e44fa8dc0f5f42fe6f915"
format_stamp: "Formatted at 2026-08-10 03:54:45 on dist-test-slave-pww0"
server_key: "ce45499e96db6b858836e2f3a1c1c81e"
server_key_iv: "428b8e7ce04d26bb396f563d209a3f04"
server_key_version: "encryptionkey@0"
I20260810 03:54:45.549152 32393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "b4e1108fbd6e44fa8dc0f5f42fe6f915"
format_stamp: "Formatted at 2026-08-10 03:54:45 on dist-test-slave-pww0"
server_key: "ce45499e96db6b858836e2f3a1c1c81e"
server_key_iv: "428b8e7ce04d26bb396f563d209a3f04"
server_key_version: "encryptionkey@0"
I20260810 03:54:45.552868 32393 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 03:54:45.555341 32418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:45.556654 32393 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 03:54:45.556798 32393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "b4e1108fbd6e44fa8dc0f5f42fe6f915"
format_stamp: "Formatted at 2026-08-10 03:54:45 on dist-test-slave-pww0"
server_key: "ce45499e96db6b858836e2f3a1c1c81e"
server_key_iv: "428b8e7ce04d26bb396f563d209a3f04"
server_key_version: "encryptionkey@0"
I20260810 03:54:45.556906 32393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:45.578033 32393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:45.582463 32393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:45.582696 32393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:45.594583 32393 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33441
I20260810 03:54:45.594647 32521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33441 every 8 connection(s)
I20260810 03:54:45.595924 32393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:54:45.597798 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 32393
I20260810 03:54:45.597932 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:54:45.598235 28261 external_mini_cluster.cc:1428] Setting key e46f63b4bcf141afa21cc8d98bebe234
I20260810 03:54:45.599655 32522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:45.606678 32522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4e1108fbd6e44fa8dc0f5f42fe6f915: Bootstrap starting.
I20260810 03:54:45.609186 32522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4e1108fbd6e44fa8dc0f5f42fe6f915: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:45.610154 32522 log.cc:824] T 00000000000000000000000000000000 P b4e1108fbd6e44fa8dc0f5f42fe6f915: Log is configured to *not* fsync() on all Append() calls
Aug 10 03:54:45 dist-test-slave-pww0 krb5kdc[32369](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 03:54:45.614756 32522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4e1108fbd6e44fa8dc0f5f42fe6f915: No bootstrap required, opened a new log
I20260810 03:54:45.619541 32522 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4e1108fbd6e44fa8dc0f5f42fe6f915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e1108fbd6e44fa8dc0f5f42fe6f915" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33441 } }
I20260810 03:54:45.619854 32522 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4e1108fbd6e44fa8dc0f5f42fe6f915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:45.619927 32522 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4e1108fbd6e44fa8dc0f5f42fe6f915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4e1108fbd6e44fa8dc0f5f42fe6f915, State: Initialized, Role: FOLLOWER
I20260810 03:54:45.620518 32522 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4e1108fbd6e44fa8dc0f5f42fe6f915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e1108fbd6e44fa8dc0f5f42fe6f915" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33441 } }
I20260810 03:54:45.620697 32522 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4e1108fbd6e44fa8dc0f5f42fe6f915 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:54:45.620764 32522 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4e1108fbd6e44fa8dc0f5f42fe6f915 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:54:45.620865 32522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4e1108fbd6e44fa8dc0f5f42fe6f915 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:45.621284 32526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:45.600037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39363 (local address 127.0.0.1:33441)
0810 03:54:45.600647 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:45.600668 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:45.600713 (+ 45us) server_negotiation.cc:407] Connection header received
0810 03:54:45.601605 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:45.601625 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:45.602043 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:45.602448 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:45.603919 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:45.605222 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:45.610730 (+ 5508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:45.611609 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:45.611798 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:45.614993 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:45.615002 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:45.615024 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:45.615062 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:45.617607 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:45.618271 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:45.618279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:45.618286 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:45.618445 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:45.618822 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:45.618825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:45.618827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:45.619371 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:45.619629 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:45.619964 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 03:54:45.620205 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":171,"threads_started":1}
I20260810 03:54:45.622058 32522 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4e1108fbd6e44fa8dc0f5f42fe6f915 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e1108fbd6e44fa8dc0f5f42fe6f915" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33441 } }
I20260810 03:54:45.622422 32522 leader_election.cc:302] T 00000000000000000000000000000000 P b4e1108fbd6e44fa8dc0f5f42fe6f915 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4e1108fbd6e44fa8dc0f5f42fe6f915; no voters:
I20260810 03:54:45.622786 32522 leader_election.cc:288] T 00000000000000000000000000000000 P b4e1108fbd6e44fa8dc0f5f42fe6f915 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:54:45.622915 32528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4e1108fbd6e44fa8dc0f5f42fe6f915 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:54:45.623148 32528 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4e1108fbd6e44fa8dc0f5f42fe6f915 [term 1 LEADER]: Becoming Leader. State: Replica: b4e1108fbd6e44fa8dc0f5f42fe6f915, State: Running, Role: LEADER
I20260810 03:54:45.624054 32528 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4e1108fbd6e44fa8dc0f5f42fe6f915 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e1108fbd6e44fa8dc0f5f42fe6f915" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33441 } }
I20260810 03:54:45.624148 32522 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4e1108fbd6e44fa8dc0f5f42fe6f915 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:54:45.624511 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:54:45.627698 32528 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e1108fbd6e44fa8dc0f5f42fe6f915 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4e1108fbd6e44fa8dc0f5f42fe6f915. Latest consensus state: current_term: 1 leader_uuid: "b4e1108fbd6e44fa8dc0f5f42fe6f915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e1108fbd6e44fa8dc0f5f42fe6f915" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33441 } } }
I20260810 03:54:45.627841 32528 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e1108fbd6e44fa8dc0f5f42fe6f915 [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:45.628733 32534 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e1108fbd6e44fa8dc0f5f42fe6f915 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4e1108fbd6e44fa8dc0f5f42fe6f915" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e1108fbd6e44fa8dc0f5f42fe6f915" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33441 } } }
I20260810 03:54:45.629135 32534 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e1108fbd6e44fa8dc0f5f42fe6f915 [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:45.630466 32564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:54:45.633837 32564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:54:45.640547 32564 catalog_manager.cc:1347] Generated new cluster ID: a7964cb7aed2474b85d1c6775898d0fc
I20260810 03:54:45.640628 32564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:54:45.641053 32526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:45.627938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48967 (local address 127.0.0.1:33441)
0810 03:54:45.628059 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:45.628064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:45.628730 (+ 666us) server_negotiation.cc:407] Connection header received
0810 03:54:45.629164 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:45.629167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:45.629218 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:45.629282 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:45.630236 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:45.631016 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:45.632296 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:45.632662 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:45.632746 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:45.636991 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:45.636996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:45.637000 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:45.637035 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:45.638763 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:45.639355 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:45.639359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:45.639361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:45.639496 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:45.639857 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:45.639861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:45.639863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:45.640096 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:45.640247 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:45.640572 (+ 325us) server_negotiation.cc:299] Negotiation successful
0810 03:54:45.640684 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 03:54:45.663733 32564 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:54:45.665076 32564 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:54:45.682945 32564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4e1108fbd6e44fa8dc0f5f42fe6f915: Generated new TSK 0
I20260810 03:54:45.683914 32564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:54:45.696079 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 32393
W20260810 03:54:45.714077 32367 connection.cc:535] client connection to 127.0.0.1:33441 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33441): Connection reset by peer (error 104)
2026-08-10T03:54:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1636 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:54:45 dist-test-slave-pww0 krb5kdc[32586](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:54:45 dist-test-slave-pww0 krb5kdc[32586](info): set up 2 sockets
Aug 10 03:54:45 dist-test-slave-pww0 krb5kdc[32586](info): commencing operation
krb5kdc: starting...
W20260810 03:54:46.899830 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:54:46 dist-test-slave-pww0 krb5kdc[32586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:54:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:54:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:54:47.030510 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35807
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:54:47.149251 32611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:47.149562 32611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:47.149606 32611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:47.153808 32611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:54:47.153889 32611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:47.153909 32611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:47.153925 32611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:54:47.153940 32611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:54:47.159655 32611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35807
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:47.160835 32611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:47.161907 32611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:54:47.171008 32611 server_base.cc:1034] running on GCE node
W20260810 03:54:47.170950 32636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:47.170953 32622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:47.170953 32628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:47.171875 32611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:47.173040 32611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:47.174206 32611 hybrid_clock.cc:648] HybridClock initialized: now 1786334087174186 us; error 37 us; skew 500 ppm
Aug 10 03:54:47 dist-test-slave-pww0 krb5kdc[32586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:47.177348 32611 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:54:47.178376 32611 webserver.cc:460] Webserver started at http://127.0.0.1:45121/ using document root <none> and password file <none>
I20260810 03:54:47.178952 32611 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:47.179023 32611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:47.179251 32611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:47.181556 32611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "71af47457b0d4b6794ccbaf3b1894865"
format_stamp: "Formatted at 2026-08-10 03:54:47 on dist-test-slave-pww0"
server_key: "8c3345c611986f3b06da7fd1e9002555"
server_key_iv: "64bba40f66004741fb4490a6776ee2eb"
server_key_version: "encryptionkey@0"
I20260810 03:54:47.182161 32611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "71af47457b0d4b6794ccbaf3b1894865"
format_stamp: "Formatted at 2026-08-10 03:54:47 on dist-test-slave-pww0"
server_key: "8c3345c611986f3b06da7fd1e9002555"
server_key_iv: "64bba40f66004741fb4490a6776ee2eb"
server_key_version: "encryptionkey@0"
I20260810 03:54:47.186288 32611 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:54:47.189023 32649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:47.190505 32611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:54:47.190642 32611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "71af47457b0d4b6794ccbaf3b1894865"
format_stamp: "Formatted at 2026-08-10 03:54:47 on dist-test-slave-pww0"
server_key: "8c3345c611986f3b06da7fd1e9002555"
server_key_iv: "64bba40f66004741fb4490a6776ee2eb"
server_key_version: "encryptionkey@0"
I20260810 03:54:47.190755 32611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:47.210770 32611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:47.213982 32611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:47.214190 32611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:47.221680 32611 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35807
I20260810 03:54:47.221688 32749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35807 every 8 connection(s)
I20260810 03:54:47.222695 32611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:54:47.225531 32750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:47.228438 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 32611
I20260810 03:54:47.228588 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:54:47.228924 28261 external_mini_cluster.cc:1428] Setting key a6196fec3bb245112cf055fbc32a0f7f
I20260810 03:54:47.231941 32750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865: Bootstrap starting.
I20260810 03:54:47.234233 32750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:47.235006 32750 log.cc:824] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:47.237111 32750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865: No bootstrap required, opened a new log
I20260810 03:54:47.238148 32755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:47.230417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48567 (local address 127.0.0.1:35807)
0810 03:54:47.230910 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:47.230926 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:47.230963 (+ 37us) server_negotiation.cc:407] Connection header received
0810 03:54:47.231648 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:47.231667 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:47.232045 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:47.232383 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:47.232814 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:47.233813 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:47.234774 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:47.235287 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:47.235439 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:47.236387 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:47.236393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:47.236406 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:47.236817 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:47.236876 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:47.236901 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:47.237022 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:47.237209 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 03:54:47.237371 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":147,"threads_started":1}
I20260810 03:54:47.240886 32750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71af47457b0d4b6794ccbaf3b1894865" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35807 } }
I20260810 03:54:47.241185 32750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:47.241178 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:54:47.241248 32750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71af47457b0d4b6794ccbaf3b1894865, State: Initialized, Role: FOLLOWER
I20260810 03:54:47.241729 32750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71af47457b0d4b6794ccbaf3b1894865" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35807 } }
I20260810 03:54:47.241889 32750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:54:47.241955 32750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:54:47.242058 32750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:47.243371 32750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71af47457b0d4b6794ccbaf3b1894865" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35807 } }
I20260810 03:54:47.243798 32750 leader_election.cc:302] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71af47457b0d4b6794ccbaf3b1894865; no voters:
I20260810 03:54:47.244164 32750 leader_election.cc:288] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:54:47.244266 301 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:54:47.244499 301 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865 [term 1 LEADER]: Becoming Leader. State: Replica: 71af47457b0d4b6794ccbaf3b1894865, State: Running, Role: LEADER
I20260810 03:54:47.244839 301 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71af47457b0d4b6794ccbaf3b1894865" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35807 } }
I20260810 03:54:47.245512 32750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:54:47.247328 304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71af47457b0d4b6794ccbaf3b1894865. Latest consensus state: current_term: 1 leader_uuid: "71af47457b0d4b6794ccbaf3b1894865" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71af47457b0d4b6794ccbaf3b1894865" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35807 } } }
I20260810 03:54:47.247498 304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865 [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:47.247850 302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71af47457b0d4b6794ccbaf3b1894865" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71af47457b0d4b6794ccbaf3b1894865" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35807 } } }
I20260810 03:54:47.247949 302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865 [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:47.249235 32755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:47.243551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50585 (local address 127.0.0.1:35807)
0810 03:54:47.243686 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:47.243691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:47.243969 (+ 278us) server_negotiation.cc:407] Connection header received
0810 03:54:47.244166 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:47.244170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:47.244225 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:47.244304 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:47.244795 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:47.245568 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:47.246738 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:47.247062 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:47.247162 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:47.248841 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:47.248844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:47.248846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:47.248859 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:47.248885 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:47.248895 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:47.248959 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:47.249098 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 03:54:47.249131 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260810 03:54:47.251049 332 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:54:47.251118 332 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:54:47.251192 330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:54:47.251979 330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:54:47.257308 330 catalog_manager.cc:1347] Generated new cluster ID: 8500d74980ba469fa0a18c416eaf108e
I20260810 03:54:47.257392 330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:54:47.270824 330 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:54:47.272326 330 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:54:47.281311 330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71af47457b0d4b6794ccbaf3b1894865: Generated new TSK 0
I20260810 03:54:47.282114 330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:54:47.285953 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 32611
W20260810 03:54:47.300042 32579 connection.cc:535] client connection to 127.0.0.1:35807 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35807): Connection reset by peer (error 104)
2026-08-10T03:54:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1586 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:54:47 dist-test-slave-pww0 krb5kdc[339](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:54:47 dist-test-slave-pww0 krb5kdc[339](info): set up 2 sockets
Aug 10 03:54:47 dist-test-slave-pww0 krb5kdc[339](info): commencing operation
krb5kdc: starting...
W20260810 03:54:48.471619 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:54:48 dist-test-slave-pww0 krb5kdc[339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:54:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:54:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:54:48.602365 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38109
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:54:48.718888 360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:48.719188 360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:48.719257 360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:48.723383 360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:54:48.723495 360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:48.723526 360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:48.723556 360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:54:48.723585 360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:54:48.729338 360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38109
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:48.730545 360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:48.731634 360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:54:48.740063 360 server_base.cc:1034] running on GCE node
W20260810 03:54:48.740012 374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:48.740012 373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:48.740229 376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:48.740859 360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:48.741988 360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:48.743148 360 hybrid_clock.cc:648] HybridClock initialized: now 1786334088743131 us; error 29 us; skew 500 ppm
Aug 10 03:54:48 dist-test-slave-pww0 krb5kdc[339](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:48.749225 360 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:54:48.750212 360 webserver.cc:460] Webserver started at http://127.0.0.1:46487/ using document root <none> and password file <none>
I20260810 03:54:48.750800 360 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:48.750870 360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:48.751080 360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:48.753274 360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "ffd6b2da0d4c48b7a2f053a13139a687"
format_stamp: "Formatted at 2026-08-10 03:54:48 on dist-test-slave-pww0"
server_key: "fae1820b6bf9054732ca314d0ce17f49"
server_key_iv: "6a26a0e085dd65e2522cc4d75dd2f438"
server_key_version: "encryptionkey@0"
I20260810 03:54:48.753865 360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "ffd6b2da0d4c48b7a2f053a13139a687"
format_stamp: "Formatted at 2026-08-10 03:54:48 on dist-test-slave-pww0"
server_key: "fae1820b6bf9054732ca314d0ce17f49"
server_key_iv: "6a26a0e085dd65e2522cc4d75dd2f438"
server_key_version: "encryptionkey@0"
I20260810 03:54:48.757733 360 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 03:54:48.760279 399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:48.761495 360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:54:48.761620 360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "ffd6b2da0d4c48b7a2f053a13139a687"
format_stamp: "Formatted at 2026-08-10 03:54:48 on dist-test-slave-pww0"
server_key: "fae1820b6bf9054732ca314d0ce17f49"
server_key_iv: "6a26a0e085dd65e2522cc4d75dd2f438"
server_key_version: "encryptionkey@0"
I20260810 03:54:48.761720 360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:48.766435 360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:48.769268 360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:48.769438 360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:48.776381 360 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38109
I20260810 03:54:48.776398 490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38109 every 8 connection(s)
I20260810 03:54:48.777328 360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:54:48.779861 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 360
I20260810 03:54:48.779999 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:54:48.780311 28261 external_mini_cluster.cc:1428] Setting key d0cba82141d32f6d18e01b6726cb5563
I20260810 03:54:48.780409 491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:48.785705 496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:48.781665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56875 (local address 127.0.0.1:38109)
0810 03:54:48.782073 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:48.782089 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:48.782124 (+ 35us) server_negotiation.cc:407] Connection header received
0810 03:54:48.782770 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:48.782787 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:48.783077 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:48.783375 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:48.784403 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:48.784408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:48.784422 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:48.784433 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:48.784494 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:48.784514 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:48.784632 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:48.784788 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 03:54:48.784948 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":117,"threads_started":1}
I20260810 03:54:48.787849 491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687: Bootstrap starting.
I20260810 03:54:48.788761 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:54:48.790719 491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:48.791689 491 log.cc:824] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:48.792963 496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:48.791158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36333 (local address 127.0.0.1:38109)
0810 03:54:48.791289 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:48.791293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:48.791581 (+ 288us) server_negotiation.cc:407] Connection header received
0810 03:54:48.791728 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:48.791731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:48.791774 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:48.791850 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:48.792596 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:48.792599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:48.792601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:48.792611 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:48.792636 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:48.792646 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:48.792712 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:48.792813 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 03:54:48.792845 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 03:54:48.793995 491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687: No bootstrap required, opened a new log
I20260810 03:54:48.797713 491 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd6b2da0d4c48b7a2f053a13139a687" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38109 } }
I20260810 03:54:48.797971 491 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:48.798023 491 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd6b2da0d4c48b7a2f053a13139a687, State: Initialized, Role: FOLLOWER
I20260810 03:54:48.798475 491 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd6b2da0d4c48b7a2f053a13139a687" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38109 } }
I20260810 03:54:48.798617 491 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:54:48.798686 491 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:54:48.798790 491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:48.799829 491 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd6b2da0d4c48b7a2f053a13139a687" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38109 } }
I20260810 03:54:48.800173 491 leader_election.cc:302] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffd6b2da0d4c48b7a2f053a13139a687; no voters:
I20260810 03:54:48.800486 491 leader_election.cc:288] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:54:48.800606 509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:54:48.800866 509 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687 [term 1 LEADER]: Becoming Leader. State: Replica: ffd6b2da0d4c48b7a2f053a13139a687, State: Running, Role: LEADER
I20260810 03:54:48.801168 509 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd6b2da0d4c48b7a2f053a13139a687" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38109 } }
I20260810 03:54:48.801769 491 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:54:48.802862 512 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffd6b2da0d4c48b7a2f053a13139a687" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd6b2da0d4c48b7a2f053a13139a687" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38109 } } }
I20260810 03:54:48.803014 512 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687 [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:48.803359 513 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffd6b2da0d4c48b7a2f053a13139a687. Latest consensus state: current_term: 1 leader_uuid: "ffd6b2da0d4c48b7a2f053a13139a687" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd6b2da0d4c48b7a2f053a13139a687" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38109 } } }
I20260810 03:54:48.803485 513 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687 [sys.catalog]: This master's current role is: LEADER
W20260810 03:54:48.806871 539 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:54:48.806965 539 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:54:48.807042 524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:54:48.807844 524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:54:48.813369 524 catalog_manager.cc:1347] Generated new cluster ID: fd39801ee94f4e67850f192c85bca445
I20260810 03:54:48.813453 524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:54:48.820355 524 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:54:48.820734 524 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:54:48.838553 524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffd6b2da0d4c48b7a2f053a13139a687: Generated new TSK 0
I20260810 03:54:48.839560 524 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:54:48.848086 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 360
2026-08-10T03:54:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1558 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:54:48 dist-test-slave-pww0 krb5kdc[552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:54:48 dist-test-slave-pww0 krb5kdc[552](info): set up 2 sockets
Aug 10 03:54:48 dist-test-slave-pww0 krb5kdc[552](info): commencing operation
krb5kdc: starting...
W20260810 03:54:50.046319 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:54:50 dist-test-slave-pww0 krb5kdc[552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:54:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:54:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:54:50.175843 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43603
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:54:50.292728 576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:50.293035 576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:50.293084 576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:50.297358 576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:54:50.297469 576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:50.297515 576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:50.297552 576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:54:50.297573 576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:54:50.303244 576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:50.304467 576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:50.305519 576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:50.314422 586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:50.314432 601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:50.314503 576 server_base.cc:1034] running on GCE node
W20260810 03:54:50.314432 592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:50.315328 576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:50.316558 576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:50.317775 576 hybrid_clock.cc:648] HybridClock initialized: now 1786334090317734 us; error 38 us; skew 500 ppm
Aug 10 03:54:50 dist-test-slave-pww0 krb5kdc[552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:50.320684 576 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:54:50.321695 576 webserver.cc:460] Webserver started at http://127.0.0.1:41573/ using document root <none> and password file <none>
I20260810 03:54:50.322263 576 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:50.322338 576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:50.322558 576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:50.324745 576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "3aef4a0cd24340a6aa211d57733d275b"
format_stamp: "Formatted at 2026-08-10 03:54:50 on dist-test-slave-pww0"
server_key: "97260b3e73c06649ec357c7e571782c6"
server_key_iv: "08c91c9feaa3cb487d073a507133d06a"
server_key_version: "encryptionkey@0"
I20260810 03:54:50.325326 576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "3aef4a0cd24340a6aa211d57733d275b"
format_stamp: "Formatted at 2026-08-10 03:54:50 on dist-test-slave-pww0"
server_key: "97260b3e73c06649ec357c7e571782c6"
server_key_iv: "08c91c9feaa3cb487d073a507133d06a"
server_key_version: "encryptionkey@0"
I20260810 03:54:50.329315 576 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:54:50.332015 613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:50.333436 576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:54:50.333598 576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "3aef4a0cd24340a6aa211d57733d275b"
format_stamp: "Formatted at 2026-08-10 03:54:50 on dist-test-slave-pww0"
server_key: "97260b3e73c06649ec357c7e571782c6"
server_key_iv: "08c91c9feaa3cb487d073a507133d06a"
server_key_version: "encryptionkey@0"
I20260810 03:54:50.333763 576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:50.343662 576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:50.346972 576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:50.347173 576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:50.354384 576 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43603
I20260810 03:54:50.354432 717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43603 every 8 connection(s)
I20260810 03:54:50.355424 576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:54:50.358459 719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:50.364024 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 576
I20260810 03:54:50.364176 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:54:50.364462 28261 external_mini_cluster.cc:1428] Setting key bd0c211459ea4c63c61f56547d3da8ec
I20260810 03:54:50.364769 719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aef4a0cd24340a6aa211d57733d275b: Bootstrap starting.
I20260810 03:54:50.367527 719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aef4a0cd24340a6aa211d57733d275b: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:50.368317 719 log.cc:824] T 00000000000000000000000000000000 P 3aef4a0cd24340a6aa211d57733d275b: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:50.370386 719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aef4a0cd24340a6aa211d57733d275b: No bootstrap required, opened a new log
Aug 10 03:54:50 dist-test-slave-pww0 krb5kdc[552](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 03:54:50.374090 719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aef4a0cd24340a6aa211d57733d275b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aef4a0cd24340a6aa211d57733d275b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43603 } }
I20260810 03:54:50.374368 719 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3aef4a0cd24340a6aa211d57733d275b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:50.374429 719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aef4a0cd24340a6aa211d57733d275b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aef4a0cd24340a6aa211d57733d275b, State: Initialized, Role: FOLLOWER
I20260810 03:54:50.375031 719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aef4a0cd24340a6aa211d57733d275b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aef4a0cd24340a6aa211d57733d275b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43603 } }
I20260810 03:54:50.375216 719 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3aef4a0cd24340a6aa211d57733d275b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:54:50.375298 719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aef4a0cd24340a6aa211d57733d275b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:54:50.375414 719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aef4a0cd24340a6aa211d57733d275b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:50.376646 719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aef4a0cd24340a6aa211d57733d275b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aef4a0cd24340a6aa211d57733d275b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43603 } }
I20260810 03:54:50.377043 719 leader_election.cc:302] T 00000000000000000000000000000000 P 3aef4a0cd24340a6aa211d57733d275b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3aef4a0cd24340a6aa211d57733d275b; no voters:
I20260810 03:54:50.377441 719 leader_election.cc:288] T 00000000000000000000000000000000 P 3aef4a0cd24340a6aa211d57733d275b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:54:50.377548 726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3aef4a0cd24340a6aa211d57733d275b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:54:50.377789 726 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3aef4a0cd24340a6aa211d57733d275b [term 1 LEADER]: Becoming Leader. State: Replica: 3aef4a0cd24340a6aa211d57733d275b, State: Running, Role: LEADER
I20260810 03:54:50.378115 726 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3aef4a0cd24340a6aa211d57733d275b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aef4a0cd24340a6aa211d57733d275b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43603 } }
I20260810 03:54:50.378894 719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aef4a0cd24340a6aa211d57733d275b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:54:50.379879 732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aef4a0cd24340a6aa211d57733d275b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aef4a0cd24340a6aa211d57733d275b. Latest consensus state: current_term: 1 leader_uuid: "3aef4a0cd24340a6aa211d57733d275b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aef4a0cd24340a6aa211d57733d275b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43603 } } }
I20260810 03:54:50.379879 731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aef4a0cd24340a6aa211d57733d275b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3aef4a0cd24340a6aa211d57733d275b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aef4a0cd24340a6aa211d57733d275b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43603 } } }
I20260810 03:54:50.380031 732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aef4a0cd24340a6aa211d57733d275b [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:50.380033 731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aef4a0cd24340a6aa211d57733d275b [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:50.380262 722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:50.365988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40781 (local address 127.0.0.1:43603)
0810 03:54:50.366446 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:50.366461 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:50.366498 (+ 37us) server_negotiation.cc:407] Connection header received
0810 03:54:50.367220 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:50.367240 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:50.367678 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:50.368024 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:50.368897 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:50.369931 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:50.370885 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:50.371381 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:50.371571 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:50.374000 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:50.374005 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:50.374022 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:50.374053 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:50.376199 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:50.376662 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:50.376669 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:50.376675 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:50.376846 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:50.377115 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:50.377118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:50.377120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:50.377684 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:50.378634 (+ 950us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:50.378805 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 03:54:50.379130 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":141,"threads_started":1}
I20260810 03:54:50.380442 746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:54:50.383853 746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:54:50.383870 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:54:50.390690 746 catalog_manager.cc:1347] Generated new cluster ID: b47e8c86b05044cca2dfd2ec26fcba8b
I20260810 03:54:50.390763 746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:54:50.396098 722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:50.386292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49801 (local address 127.0.0.1:43603)
0810 03:54:50.386411 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:50.386415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:50.386752 (+ 337us) server_negotiation.cc:407] Connection header received
0810 03:54:50.386873 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:50.386876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:50.386931 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:50.387017 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:50.387916 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:50.388741 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:50.389629 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:50.389884 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:50.389975 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:50.392518 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:50.392522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:50.392525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:50.392555 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:50.394247 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:50.394780 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:50.394784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:50.394787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:50.394888 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:50.395135 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:50.395138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:50.395140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:50.395378 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:50.395556 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:50.395772 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 03:54:50.395910 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 03:54:50.399654 746 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:54:50.400604 746 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:54:50.407371 746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3aef4a0cd24340a6aa211d57733d275b: Generated new TSK 0
I20260810 03:54:50.408140 746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:54:50.413746 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 576
2026-08-10T03:54:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1567 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:54:50 dist-test-slave-pww0 krb5kdc[780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:54:50 dist-test-slave-pww0 krb5kdc[780](info): set up 2 sockets
Aug 10 03:54:50 dist-test-slave-pww0 krb5kdc[780](info): commencing operation
krb5kdc: starting...
W20260810 03:54:51.616319 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:54:51 dist-test-slave-pww0 krb5kdc[780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:54:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:54:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:54:51.750043 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39963
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:54:51.868543 804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:51.868865 804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:51.868928 804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:51.873098 804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:54:51.873178 804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:51.873204 804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:51.873227 804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:54:51.873255 804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:54:51.878902 804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:51.880167 804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:51.881237 804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:51.890105 821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:51.890105 815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:51.890105 817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:51.890316 804 server_base.cc:1034] running on GCE node
I20260810 03:54:51.891042 804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:51.892309 804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:51.893498 804 hybrid_clock.cc:648] HybridClock initialized: now 1786334091893474 us; error 31 us; skew 500 ppm
Aug 10 03:54:51 dist-test-slave-pww0 krb5kdc[780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:51.896432 804 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:54:51.897440 804 webserver.cc:460] Webserver started at http://127.0.0.1:44281/ using document root <none> and password file <none>
I20260810 03:54:51.898020 804 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:51.898093 804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:51.898314 804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:51.900542 804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "9baa1f84abd54d2f93780c1d89634b17"
format_stamp: "Formatted at 2026-08-10 03:54:51 on dist-test-slave-pww0"
server_key: "8b2352750fa202846dabb82cb8dbda25"
server_key_iv: "83e46eae7d580f53ae9a63661c4eb955"
server_key_version: "encryptionkey@0"
I20260810 03:54:51.901121 804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "9baa1f84abd54d2f93780c1d89634b17"
format_stamp: "Formatted at 2026-08-10 03:54:51 on dist-test-slave-pww0"
server_key: "8b2352750fa202846dabb82cb8dbda25"
server_key_iv: "83e46eae7d580f53ae9a63661c4eb955"
server_key_version: "encryptionkey@0"
I20260810 03:54:51.905181 804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:54:51.907963 835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:51.909373 804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:54:51.909518 804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "9baa1f84abd54d2f93780c1d89634b17"
format_stamp: "Formatted at 2026-08-10 03:54:51 on dist-test-slave-pww0"
server_key: "8b2352750fa202846dabb82cb8dbda25"
server_key_iv: "83e46eae7d580f53ae9a63661c4eb955"
server_key_version: "encryptionkey@0"
I20260810 03:54:51.909636 804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:51.925274 804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:51.928937 804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:51.929152 804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:51.936970 804 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39963
I20260810 03:54:51.936986 928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39963 every 8 connection(s)
I20260810 03:54:51.937971 804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:54:51.940996 929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:51.947166 929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17: Bootstrap starting.
I20260810 03:54:51.947907 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 804
I20260810 03:54:51.948057 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:54:51.948350 28261 external_mini_cluster.cc:1428] Setting key a109785f258828ae4781920692f1f00f
I20260810 03:54:51.950440 929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:51.951241 929 log.cc:824] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:51.953433 929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17: No bootstrap required, opened a new log
I20260810 03:54:51.957746 929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9baa1f84abd54d2f93780c1d89634b17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39963 } }
I20260810 03:54:51.958003 929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:51.958039 929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9baa1f84abd54d2f93780c1d89634b17, State: Initialized, Role: FOLLOWER
I20260810 03:54:51.958175 936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:51.949838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43925 (local address 127.0.0.1:39963)
0810 03:54:51.950375 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:51.950391 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:51.950431 (+ 40us) server_negotiation.cc:407] Connection header received
0810 03:54:51.951179 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:51.951199 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:51.951644 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:51.952046 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:51.952424 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:51.953413 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:51.954650 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:51.955170 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:51.955293 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:51.956265 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:51.956271 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:51.956288 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:51.956712 (+ 424us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:51.956772 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:51.956797 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:51.956878 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:51.957061 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 03:54:51.957316 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":152,"threads_started":1}
I20260810 03:54:51.958540 929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9baa1f84abd54d2f93780c1d89634b17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39963 } }
I20260810 03:54:51.958710 929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:54:51.958773 929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:54:51.958881 929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:51.960012 929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9baa1f84abd54d2f93780c1d89634b17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39963 } }
I20260810 03:54:51.960417 929 leader_election.cc:302] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9baa1f84abd54d2f93780c1d89634b17; no voters:
I20260810 03:54:51.960772 929 leader_election.cc:288] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:54:51.960876 939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:54:51.961387 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:54:51.961453 939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17 [term 1 LEADER]: Becoming Leader. State: Replica: 9baa1f84abd54d2f93780c1d89634b17, State: Running, Role: LEADER
I20260810 03:54:51.961772 939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9baa1f84abd54d2f93780c1d89634b17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39963 } }
I20260810 03:54:51.962095 929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:54:51.963579 942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9baa1f84abd54d2f93780c1d89634b17. Latest consensus state: current_term: 1 leader_uuid: "9baa1f84abd54d2f93780c1d89634b17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9baa1f84abd54d2f93780c1d89634b17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39963 } } }
I20260810 03:54:51.963733 942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17 [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:51.964047 941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9baa1f84abd54d2f93780c1d89634b17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9baa1f84abd54d2f93780c1d89634b17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39963 } } }
I20260810 03:54:51.964128 941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17 [sys.catalog]: This master's current role is: LEADER
W20260810 03:54:51.967195 974 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:54:51.967283 974 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:54:51.967403 969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:54:51.968351 969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:54:51.968510 936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:51.963801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36767 (local address 127.0.0.1:39963)
0810 03:54:51.963910 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:51.963915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:51.964218 (+ 303us) server_negotiation.cc:407] Connection header received
0810 03:54:51.964410 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:51.964414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:51.964457 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:51.964503 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:51.965006 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:51.965739 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:51.966848 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:51.967068 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:51.967159 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:51.967994 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:51.967998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:51.968001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:51.968015 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:51.968045 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:51.968058 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:51.968138 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:51.968263 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 03:54:51.968321 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 03:54:51.974743 969 catalog_manager.cc:1347] Generated new cluster ID: 727c115cd41948ac98d31761dc669c96
I20260810 03:54:51.974915 969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:54:51.984748 969 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:54:51.986091 969 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:54:51.996970 969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9baa1f84abd54d2f93780c1d89634b17: Generated new TSK 0
I20260810 03:54:51.997926 969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:54:52.006685 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 804
2026-08-10T03:54:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1606 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:54:52 dist-test-slave-pww0 krb5kdc[985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:54:52 dist-test-slave-pww0 krb5kdc[985](info): set up 2 sockets
Aug 10 03:54:52 dist-test-slave-pww0 krb5kdc[985](info): commencing operation
krb5kdc: starting...
W20260810 03:54:53.222558 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.175s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:54:53 dist-test-slave-pww0 krb5kdc[985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:54:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:54:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:54:53.354893 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40197
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:54:53.471671 1022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:53.471989 1022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:53.472054 1022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:53.476210 1022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:54:53.476291 1022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:53.476318 1022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:53.476338 1022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:54:53.476361 1022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:54:53.482017 1022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:53.483239 1022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:53.484419 1022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:53.493882 1029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:53.493854 1027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:53.494002 1022 server_base.cc:1034] running on GCE node
W20260810 03:54:53.493854 1031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:53.494813 1022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:53.496102 1022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:53.497309 1022 hybrid_clock.cc:648] HybridClock initialized: now 1786334093497291 us; error 32 us; skew 500 ppm
Aug 10 03:54:53 dist-test-slave-pww0 krb5kdc[985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:53.500655 1022 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:54:53.501718 1022 webserver.cc:460] Webserver started at http://127.0.0.1:33953/ using document root <none> and password file <none>
I20260810 03:54:53.502344 1022 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:53.502424 1022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:53.502655 1022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:53.505016 1022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "fce6f09b8ff643b78c57a8d540881266"
format_stamp: "Formatted at 2026-08-10 03:54:53 on dist-test-slave-pww0"
server_key: "7104d9385c1b34392e3e2d5ce895cb93"
server_key_iv: "440622dc1b64a0edc2dd9755b3cd6dbb"
server_key_version: "encryptionkey@0"
I20260810 03:54:53.505699 1022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "fce6f09b8ff643b78c57a8d540881266"
format_stamp: "Formatted at 2026-08-10 03:54:53 on dist-test-slave-pww0"
server_key: "7104d9385c1b34392e3e2d5ce895cb93"
server_key_iv: "440622dc1b64a0edc2dd9755b3cd6dbb"
server_key_version: "encryptionkey@0"
I20260810 03:54:53.510022 1022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:54:53.512617 1041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:53.513918 1022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 03:54:53.514050 1022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "fce6f09b8ff643b78c57a8d540881266"
format_stamp: "Formatted at 2026-08-10 03:54:53 on dist-test-slave-pww0"
server_key: "7104d9385c1b34392e3e2d5ce895cb93"
server_key_iv: "440622dc1b64a0edc2dd9755b3cd6dbb"
server_key_version: "encryptionkey@0"
I20260810 03:54:53.514158 1022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:53.518944 1022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:53.522027 1022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:53.522215 1022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:53.529951 1022 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40197
I20260810 03:54:53.530015 1149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40197 every 8 connection(s)
I20260810 03:54:53.531026 1022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:54:53.532579 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 1022
I20260810 03:54:53.532717 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:54:53.533036 28261 external_mini_cluster.cc:1428] Setting key 5b2ef31276311e1304140776c2bfe1b9
I20260810 03:54:53.534552 1150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:53.538300 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:53.534432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39605 (local address 127.0.0.1:40197)
0810 03:54:53.534859 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:53.534878 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:53.534916 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:54:53.535616 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:53.535634 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:53.535935 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:53.536233 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:53.537080 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:53.537084 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:53.537096 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:53.537106 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:53.537160 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:53.537180 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:53.537301 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:53.537380 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 03:54:53.537527 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":136,"threads_started":1}
I20260810 03:54:53.540982 1150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fce6f09b8ff643b78c57a8d540881266: Bootstrap starting.
I20260810 03:54:53.541260 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:54:53.543951 1150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fce6f09b8ff643b78c57a8d540881266: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:53.544903 1150 log.cc:824] T 00000000000000000000000000000000 P fce6f09b8ff643b78c57a8d540881266: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:53.545379 1153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:53.543630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56121 (local address 127.0.0.1:40197)
0810 03:54:53.543773 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:53.543778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:53.543993 (+ 215us) server_negotiation.cc:407] Connection header received
0810 03:54:53.544134 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:53.544137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:53.544178 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:53.544265 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:53.544986 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:53.544990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:53.544992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:53.545006 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:53.545038 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:53.545050 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:53.545133 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:53.545209 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 03:54:53.545252 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 03:54:53.547000 1150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fce6f09b8ff643b78c57a8d540881266: No bootstrap required, opened a new log
I20260810 03:54:53.550624 1150 raft_consensus.cc:348] T 00000000000000000000000000000000 P fce6f09b8ff643b78c57a8d540881266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce6f09b8ff643b78c57a8d540881266" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40197 } }
I20260810 03:54:53.550892 1150 raft_consensus.cc:374] T 00000000000000000000000000000000 P fce6f09b8ff643b78c57a8d540881266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:53.550937 1150 raft_consensus.cc:729] T 00000000000000000000000000000000 P fce6f09b8ff643b78c57a8d540881266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fce6f09b8ff643b78c57a8d540881266, State: Initialized, Role: FOLLOWER
I20260810 03:54:53.551326 1150 consensus_queue.cc:260] T 00000000000000000000000000000000 P fce6f09b8ff643b78c57a8d540881266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce6f09b8ff643b78c57a8d540881266" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40197 } }
I20260810 03:54:53.551443 1150 raft_consensus.cc:388] T 00000000000000000000000000000000 P fce6f09b8ff643b78c57a8d540881266 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:54:53.551542 1150 raft_consensus.cc:482] T 00000000000000000000000000000000 P fce6f09b8ff643b78c57a8d540881266 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:54:53.551646 1150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fce6f09b8ff643b78c57a8d540881266 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:53.552718 1150 raft_consensus.cc:504] T 00000000000000000000000000000000 P fce6f09b8ff643b78c57a8d540881266 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce6f09b8ff643b78c57a8d540881266" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40197 } }
I20260810 03:54:53.553041 1150 leader_election.cc:302] T 00000000000000000000000000000000 P fce6f09b8ff643b78c57a8d540881266 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fce6f09b8ff643b78c57a8d540881266; no voters:
I20260810 03:54:53.553354 1150 leader_election.cc:288] T 00000000000000000000000000000000 P fce6f09b8ff643b78c57a8d540881266 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:54:53.553499 1167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fce6f09b8ff643b78c57a8d540881266 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:54:53.553771 1167 raft_consensus.cc:686] T 00000000000000000000000000000000 P fce6f09b8ff643b78c57a8d540881266 [term 1 LEADER]: Becoming Leader. State: Replica: fce6f09b8ff643b78c57a8d540881266, State: Running, Role: LEADER
I20260810 03:54:53.554080 1167 consensus_queue.cc:237] T 00000000000000000000000000000000 P fce6f09b8ff643b78c57a8d540881266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce6f09b8ff643b78c57a8d540881266" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40197 } }
I20260810 03:54:53.554557 1150 sys_catalog.cc:564] T 00000000000000000000000000000000 P fce6f09b8ff643b78c57a8d540881266 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:54:53.555753 1175 sys_catalog.cc:455] T 00000000000000000000000000000000 P fce6f09b8ff643b78c57a8d540881266 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fce6f09b8ff643b78c57a8d540881266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce6f09b8ff643b78c57a8d540881266" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40197 } } }
I20260810 03:54:53.555886 1175 sys_catalog.cc:458] T 00000000000000000000000000000000 P fce6f09b8ff643b78c57a8d540881266 [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:53.556275 1187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:54:53.556241 1176 sys_catalog.cc:455] T 00000000000000000000000000000000 P fce6f09b8ff643b78c57a8d540881266 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fce6f09b8ff643b78c57a8d540881266. Latest consensus state: current_term: 1 leader_uuid: "fce6f09b8ff643b78c57a8d540881266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fce6f09b8ff643b78c57a8d540881266" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40197 } } }
I20260810 03:54:53.556387 1176 sys_catalog.cc:458] T 00000000000000000000000000000000 P fce6f09b8ff643b78c57a8d540881266 [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:53.559437 1187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:54:53.565083 1187 catalog_manager.cc:1347] Generated new cluster ID: 45c8754e93874e5f9fc62deb76ad903d
I20260810 03:54:53.565164 1187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:54:53.573017 1187 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:54:53.573393 1187 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:54:53.585166 1187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fce6f09b8ff643b78c57a8d540881266: Generated new TSK 0
I20260810 03:54:53.586112 1187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:54:53.591738 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 1022
2026-08-10T03:54:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1579 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:54:53 dist-test-slave-pww0 krb5kdc[1209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:54:53 dist-test-slave-pww0 krb5kdc[1209](info): set up 2 sockets
Aug 10 03:54:53 dist-test-slave-pww0 krb5kdc[1209](info): commencing operation
krb5kdc: starting...
W20260810 03:54:54.798842 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:54:54 dist-test-slave-pww0 krb5kdc[1209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:54:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:54:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:54:54.932722 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44149
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:54:55.048730 1240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:55.049052 1240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:55.049119 1240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:55.053339 1240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:54:55.053422 1240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:55.053452 1240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:55.053473 1240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:54:55.053498 1240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:54:55.059175 1240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:55.060397 1240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:55.061472 1240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:55.070237 1256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:55.070266 1254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:55.070322 1258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:55.070889 1240 server_base.cc:1034] running on GCE node
I20260810 03:54:55.071535 1240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:55.072701 1240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:55.073840 1240 hybrid_clock.cc:648] HybridClock initialized: now 1786334095073816 us; error 39 us; skew 500 ppm
Aug 10 03:54:55 dist-test-slave-pww0 krb5kdc[1209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:55.081830 1240 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:54:55.082960 1240 webserver.cc:460] Webserver started at http://127.0.0.1:37981/ using document root <none> and password file <none>
I20260810 03:54:55.083572 1240 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:55.083648 1240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:55.083837 1240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:55.085996 1240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "0225c532bbba4650836329d51e8517fd"
format_stamp: "Formatted at 2026-08-10 03:54:55 on dist-test-slave-pww0"
server_key: "a38f95ff57de3649eee95c3c18cf8721"
server_key_iv: "3023940493faffa747029a7c8d9df73a"
server_key_version: "encryptionkey@0"
I20260810 03:54:55.086555 1240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "0225c532bbba4650836329d51e8517fd"
format_stamp: "Formatted at 2026-08-10 03:54:55 on dist-test-slave-pww0"
server_key: "a38f95ff57de3649eee95c3c18cf8721"
server_key_iv: "3023940493faffa747029a7c8d9df73a"
server_key_version: "encryptionkey@0"
I20260810 03:54:55.090600 1240 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 03:54:55.093355 1269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:55.094743 1240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 03:54:55.094872 1240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "0225c532bbba4650836329d51e8517fd"
format_stamp: "Formatted at 2026-08-10 03:54:55 on dist-test-slave-pww0"
server_key: "a38f95ff57de3649eee95c3c18cf8721"
server_key_iv: "3023940493faffa747029a7c8d9df73a"
server_key_version: "encryptionkey@0"
I20260810 03:54:55.095000 1240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:55.115978 1240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:55.119400 1240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:55.119653 1240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:55.127489 1240 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44149
I20260810 03:54:55.127681 1353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44149 every 8 connection(s)
I20260810 03:54:55.128837 1240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:54:55.130859 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 1240
I20260810 03:54:55.130986 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:54:55.131246 28261 external_mini_cluster.cc:1428] Setting key 89a5bfd57df41c63c4c3761632e5ad0b
I20260810 03:54:55.132439 1354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:55.138309 1354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd: Bootstrap starting.
Aug 10 03:54:55 dist-test-slave-pww0 krb5kdc[1209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 03:54:55.140823 1354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:55.141727 1354 log.cc:824] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:55.143985 1354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd: No bootstrap required, opened a new log
I20260810 03:54:55.145993 1357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:55.132770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:38197 (local address 127.0.0.1:44149)
0810 03:54:55.133244 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:55.133264 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:55.133308 (+ 44us) server_negotiation.cc:407] Connection header received
0810 03:54:55.133969 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:55.133986 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:55.134347 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:55.134669 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:55.135519 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:55.136540 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:55.137430 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:55.137911 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:55.138058 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:55.140662 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:55.140668 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:55.140682 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:55.140713 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:55.142623 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:55.143175 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:55.143181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:55.143186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:55.143312 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:55.143702 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:55.143705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:55.143706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:55.144168 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:55.144378 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:55.144744 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 03:54:55.144985 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":151,"threads_started":1}
I20260810 03:54:55.147507 1354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0225c532bbba4650836329d51e8517fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } }
I20260810 03:54:55.147750 1354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:55.147805 1354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0225c532bbba4650836329d51e8517fd, State: Initialized, Role: FOLLOWER
I20260810 03:54:55.148250 1354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0225c532bbba4650836329d51e8517fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } }
I20260810 03:54:55.148387 1354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:54:55.148450 1354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:54:55.148550 1354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:55.149171 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:54:55.149677 1354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0225c532bbba4650836329d51e8517fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } }
I20260810 03:54:55.150024 1354 leader_election.cc:302] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0225c532bbba4650836329d51e8517fd; no voters:
I20260810 03:54:55.150396 1354 leader_election.cc:288] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:54:55.150780 1367 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:54:55.150974 1367 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd [term 1 LEADER]: Becoming Leader. State: Replica: 0225c532bbba4650836329d51e8517fd, State: Running, Role: LEADER
I20260810 03:54:55.151288 1367 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0225c532bbba4650836329d51e8517fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } }
I20260810 03:54:55.151871 1354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:54:55.153054 1369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0225c532bbba4650836329d51e8517fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0225c532bbba4650836329d51e8517fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } } }
I20260810 03:54:55.153187 1369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:55.153415 1369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0225c532bbba4650836329d51e8517fd. Latest consensus state: current_term: 1 leader_uuid: "0225c532bbba4650836329d51e8517fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0225c532bbba4650836329d51e8517fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44149 } } }
I20260810 03:54:55.153497 1369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd [sys.catalog]: This master's current role is: LEADER
W20260810 03:54:55.156843 1394 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:54:55.156915 1394 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:54:55.157001 1381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:54:55.157819 1381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:54:55.160593 1357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:55.151336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:55769 (local address 127.0.0.1:44149)
0810 03:54:55.151485 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:55.151492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:55.151803 (+ 311us) server_negotiation.cc:407] Connection header received
0810 03:54:55.151925 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:55.151928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:55.151981 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:55.152048 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:55.152877 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:55.153604 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:55.154822 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:55.155062 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:55.155150 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:55.156873 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:55.156878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:55.156881 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:55.156909 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:55.158602 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:55.159214 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:55.159218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:55.159220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:55.159319 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:55.159723 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:55.159727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:55.159729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:55.159924 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:55.160062 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:55.160347 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 03:54:55.160441 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 03:54:55.164129 1381 catalog_manager.cc:1347] Generated new cluster ID: fa90628ab11c45debb9a912721b4b047
I20260810 03:54:55.164198 1381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:54:55.182025 1381 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:54:55.183019 1381 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:54:55.198761 1381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0225c532bbba4650836329d51e8517fd: Generated new TSK 0
I20260810 03:54:55.199608 1381 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:54:55.211056 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 1240
W20260810 03:54:55.229887 1207 connection.cc:535] client connection to 127.0.0.1:44149 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44149): Connection reset by peer (error 104)
2026-08-10T03:54:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1617 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:54:55 dist-test-slave-pww0 krb5kdc[1404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:54:55 dist-test-slave-pww0 krb5kdc[1404](info): set up 2 sockets
Aug 10 03:54:55 dist-test-slave-pww0 krb5kdc[1404](info): commencing operation
krb5kdc: starting...
W20260810 03:54:56.412249 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.168s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:54:56 dist-test-slave-pww0 krb5kdc[1404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:54:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:54:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:54:56.546217 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44629
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:54:56.662557 1427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:56.662878 1427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:56.662945 1427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:56.667064 1427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:54:56.667153 1427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:56.667181 1427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:56.667201 1427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:54:56.667222 1427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:54:56.672904 1427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:56.674106 1427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:56.675197 1427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:56.683965 1446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:56.683965 1439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:56.684051 1442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:56.684324 1427 server_base.cc:1034] running on GCE node
I20260810 03:54:56.684974 1427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:56.686160 1427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:56.687340 1427 hybrid_clock.cc:648] HybridClock initialized: now 1786334096687322 us; error 31 us; skew 500 ppm
Aug 10 03:54:56 dist-test-slave-pww0 krb5kdc[1404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:56.690166 1427 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:54:56.691150 1427 webserver.cc:460] Webserver started at http://127.0.0.1:40073/ using document root <none> and password file <none>
I20260810 03:54:56.691753 1427 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:56.691826 1427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:56.692008 1427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:56.694077 1427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "7a3d160a88d9495ab456ad0ab92b567e"
format_stamp: "Formatted at 2026-08-10 03:54:56 on dist-test-slave-pww0"
server_key: "90b564bb6e840bf824a2073eaac0d94c"
server_key_iv: "1d8f5a5e64ced1236c3117fb6c6fa3fb"
server_key_version: "encryptionkey@0"
I20260810 03:54:56.694618 1427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "7a3d160a88d9495ab456ad0ab92b567e"
format_stamp: "Formatted at 2026-08-10 03:54:56 on dist-test-slave-pww0"
server_key: "90b564bb6e840bf824a2073eaac0d94c"
server_key_iv: "1d8f5a5e64ced1236c3117fb6c6fa3fb"
server_key_version: "encryptionkey@0"
I20260810 03:54:56.698761 1427 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 03:54:56.701577 1460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:56.702941 1427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 03:54:56.703088 1427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "7a3d160a88d9495ab456ad0ab92b567e"
format_stamp: "Formatted at 2026-08-10 03:54:56 on dist-test-slave-pww0"
server_key: "90b564bb6e840bf824a2073eaac0d94c"
server_key_iv: "1d8f5a5e64ced1236c3117fb6c6fa3fb"
server_key_version: "encryptionkey@0"
I20260810 03:54:56.703212 1427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:56.716145 1427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:56.719118 1427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:56.719338 1427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:56.727005 1427 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44629
I20260810 03:54:56.727020 1557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44629 every 8 connection(s)
I20260810 03:54:56.728051 1427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:54:56.730985 1558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:56.734166 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 1427
I20260810 03:54:56.734313 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:54:56.734555 28261 external_mini_cluster.cc:1428] Setting key ba9f4e9144ae21d20e882d1480eaf366
I20260810 03:54:56.737264 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a3d160a88d9495ab456ad0ab92b567e: Bootstrap starting.
I20260810 03:54:56.740161 1558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a3d160a88d9495ab456ad0ab92b567e: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:56.740988 1558 log.cc:824] T 00000000000000000000000000000000 P 7a3d160a88d9495ab456ad0ab92b567e: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:56.743360 1558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a3d160a88d9495ab456ad0ab92b567e: No bootstrap required, opened a new log
I20260810 03:54:56.744635 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:56.736047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:50471 (local address 127.0.0.1:44629)
0810 03:54:56.736509 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:56.736530 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:56.736573 (+ 43us) server_negotiation.cc:407] Connection header received
0810 03:54:56.737409 (+ 836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:56.737433 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:56.737894 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:56.738266 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:56.738721 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:56.740240 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:56.741162 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:56.741814 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:56.741968 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:56.742900 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:56.742905 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:56.742919 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:56.743325 (+ 406us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:56.743383 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:56.743407 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:56.743565 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:56.743710 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 03:54:56.743865 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":124,"threads_started":1}
I20260810 03:54:56.747092 1558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a3d160a88d9495ab456ad0ab92b567e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3d160a88d9495ab456ad0ab92b567e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44629 } }
I20260810 03:54:56.747339 1558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a3d160a88d9495ab456ad0ab92b567e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:56.747407 1558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a3d160a88d9495ab456ad0ab92b567e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a3d160a88d9495ab456ad0ab92b567e, State: Initialized, Role: FOLLOWER
I20260810 03:54:56.747629 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:54:56.747896 1558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a3d160a88d9495ab456ad0ab92b567e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3d160a88d9495ab456ad0ab92b567e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44629 } }
I20260810 03:54:56.748049 1558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a3d160a88d9495ab456ad0ab92b567e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:54:56.748109 1558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a3d160a88d9495ab456ad0ab92b567e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:54:56.748211 1558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a3d160a88d9495ab456ad0ab92b567e [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:56.749511 1558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a3d160a88d9495ab456ad0ab92b567e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3d160a88d9495ab456ad0ab92b567e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44629 } }
I20260810 03:54:56.749899 1558 leader_election.cc:302] T 00000000000000000000000000000000 P 7a3d160a88d9495ab456ad0ab92b567e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a3d160a88d9495ab456ad0ab92b567e; no voters:
I20260810 03:54:56.750283 1558 leader_election.cc:288] T 00000000000000000000000000000000 P 7a3d160a88d9495ab456ad0ab92b567e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:54:56.750389 1571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a3d160a88d9495ab456ad0ab92b567e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:54:56.750624 1571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a3d160a88d9495ab456ad0ab92b567e [term 1 LEADER]: Becoming Leader. State: Replica: 7a3d160a88d9495ab456ad0ab92b567e, State: Running, Role: LEADER
I20260810 03:54:56.751001 1571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a3d160a88d9495ab456ad0ab92b567e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3d160a88d9495ab456ad0ab92b567e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44629 } }
I20260810 03:54:56.751801 1558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a3d160a88d9495ab456ad0ab92b567e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:54:56.752300 1572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a3d160a88d9495ab456ad0ab92b567e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a3d160a88d9495ab456ad0ab92b567e. Latest consensus state: current_term: 1 leader_uuid: "7a3d160a88d9495ab456ad0ab92b567e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3d160a88d9495ab456ad0ab92b567e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44629 } } }
I20260810 03:54:56.752509 1572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a3d160a88d9495ab456ad0ab92b567e [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:56.752673 1575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a3d160a88d9495ab456ad0ab92b567e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a3d160a88d9495ab456ad0ab92b567e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a3d160a88d9495ab456ad0ab92b567e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44629 } } }
I20260810 03:54:56.752794 1575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a3d160a88d9495ab456ad0ab92b567e [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:56.752965 1581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:54:56.755370 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:56.750009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:58313 (local address 127.0.0.1:44629)
0810 03:54:56.750134 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:56.750138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:56.750395 (+ 257us) server_negotiation.cc:407] Connection header received
0810 03:54:56.750589 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:56.750593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:56.750650 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:56.750731 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:56.751158 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:56.751958 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:56.753088 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:56.753433 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:56.753540 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:56.754689 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:56.754693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:56.754696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:56.754710 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:56.754741 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:56.754764 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:56.754839 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:56.755202 (+ 363us) server_negotiation.cc:299] Negotiation successful
0810 03:54:56.755240 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 03:54:56.756551 1581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:54:56.762090 1581 catalog_manager.cc:1347] Generated new cluster ID: 997b6403c26e492980e0e340d94e5f46
I20260810 03:54:56.762173 1581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:54:56.771791 1581 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:54:56.773222 1581 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:54:56.782361 1581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a3d160a88d9495ab456ad0ab92b567e: Generated new TSK 0
I20260810 03:54:56.783283 1581 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:54:56.795109 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 1427
W20260810 03:54:56.805836 1402 connection.cc:535] client connection to 127.0.0.1:44629 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44629): Connection reset by peer (error 104)
2026-08-10T03:54:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1575 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:54:56 dist-test-slave-pww0 krb5kdc[1615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:54:56 dist-test-slave-pww0 krb5kdc[1615](info): set up 2 sockets
Aug 10 03:54:56 dist-test-slave-pww0 krb5kdc[1615](info): commencing operation
krb5kdc: starting...
W20260810 03:54:57.994323 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:54:58 dist-test-slave-pww0 krb5kdc[1615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:54:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:54:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:54:58.126652 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35049
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:54:58.242545 1641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:58.242880 1641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:58.242946 1641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:58.247567 1641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:54:58.247679 1641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:58.247710 1641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:58.247732 1641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:54:58.247752 1641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:54:58.253600 1641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:58.254812 1641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:58.255961 1641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:54:58.264755 1649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:58.264925 1641 server_base.cc:1034] running on GCE node
W20260810 03:54:58.264755 1655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:58.264824 1653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:58.265643 1641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:58.266877 1641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:58.268054 1641 hybrid_clock.cc:648] HybridClock initialized: now 1786334098268031 us; error 45 us; skew 500 ppm
Aug 10 03:54:58 dist-test-slave-pww0 krb5kdc[1615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:58.271152 1641 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:54:58.272228 1641 webserver.cc:460] Webserver started at http://127.0.0.1:36573/ using document root <none> and password file <none>
I20260810 03:54:58.272809 1641 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:58.272886 1641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:58.273106 1641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:58.275251 1641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "32e172db2844417ea4bd37782e07eeab"
format_stamp: "Formatted at 2026-08-10 03:54:58 on dist-test-slave-pww0"
server_key: "97053df6834551c921a32f729f3b5f5d"
server_key_iv: "7b2a5ceb9d9ac6d7f2d4a2dc12aa78f7"
server_key_version: "encryptionkey@0"
I20260810 03:54:58.275882 1641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "32e172db2844417ea4bd37782e07eeab"
format_stamp: "Formatted at 2026-08-10 03:54:58 on dist-test-slave-pww0"
server_key: "97053df6834551c921a32f729f3b5f5d"
server_key_iv: "7b2a5ceb9d9ac6d7f2d4a2dc12aa78f7"
server_key_version: "encryptionkey@0"
I20260810 03:54:58.279820 1641 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:54:58.282470 1665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:58.283813 1641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:54:58.283944 1641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "32e172db2844417ea4bd37782e07eeab"
format_stamp: "Formatted at 2026-08-10 03:54:58 on dist-test-slave-pww0"
server_key: "97053df6834551c921a32f729f3b5f5d"
server_key_iv: "7b2a5ceb9d9ac6d7f2d4a2dc12aa78f7"
server_key_version: "encryptionkey@0"
I20260810 03:54:58.284051 1641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:58.289077 1641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:58.292311 1641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:58.292505 1641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:58.300041 1641 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35049
I20260810 03:54:58.300057 1749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35049 every 8 connection(s)
I20260810 03:54:58.301040 1641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:54:58.304067 1750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:58.304358 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 1641
I20260810 03:54:58.304488 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:54:58.304790 28261 external_mini_cluster.cc:1428] Setting key bd2f17dca96f7be30b890558b5117577
I20260810 03:54:58.310386 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:58.306288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:55955 (local address 127.0.0.1:35049)
0810 03:54:58.306716 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:58.306734 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:58.306772 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:54:58.307555 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:58.307579 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:58.307891 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:58.308191 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:58.309107 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:58.309112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:58.309124 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:58.309135 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:58.309191 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:58.309210 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:58.309334 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:58.309433 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 03:54:58.309599 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":118,"threads_started":1}
I20260810 03:54:58.311444 1750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32e172db2844417ea4bd37782e07eeab: Bootstrap starting.
I20260810 03:54:58.313573 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:54:58.314267 1750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32e172db2844417ea4bd37782e07eeab: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:58.315253 1750 log.cc:824] T 00000000000000000000000000000000 P 32e172db2844417ea4bd37782e07eeab: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:58.317688 1750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32e172db2844417ea4bd37782e07eeab: No bootstrap required, opened a new log
I20260810 03:54:58.317952 1753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:58.316143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:43063 (local address 127.0.0.1:35049)
0810 03:54:58.316256 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:58.316260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:58.316467 (+ 207us) server_negotiation.cc:407] Connection header received
0810 03:54:58.316611 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:58.316614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:58.316652 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:58.316715 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:58.317539 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:58.317543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:58.317545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:54:58.317557 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:58.317584 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:54:58.317594 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:58.317670 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:58.317787 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 03:54:58.317831 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 03:54:58.321709 1750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32e172db2844417ea4bd37782e07eeab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e172db2844417ea4bd37782e07eeab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35049 } }
I20260810 03:54:58.322018 1750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32e172db2844417ea4bd37782e07eeab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:58.322088 1750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32e172db2844417ea4bd37782e07eeab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e172db2844417ea4bd37782e07eeab, State: Initialized, Role: FOLLOWER
I20260810 03:54:58.322530 1750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32e172db2844417ea4bd37782e07eeab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e172db2844417ea4bd37782e07eeab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35049 } }
I20260810 03:54:58.322659 1750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32e172db2844417ea4bd37782e07eeab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:54:58.322723 1750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32e172db2844417ea4bd37782e07eeab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:54:58.322822 1750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32e172db2844417ea4bd37782e07eeab [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:58.323918 1750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32e172db2844417ea4bd37782e07eeab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e172db2844417ea4bd37782e07eeab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35049 } }
I20260810 03:54:58.324283 1750 leader_election.cc:302] T 00000000000000000000000000000000 P 32e172db2844417ea4bd37782e07eeab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32e172db2844417ea4bd37782e07eeab; no voters:
I20260810 03:54:58.324608 1750 leader_election.cc:288] T 00000000000000000000000000000000 P 32e172db2844417ea4bd37782e07eeab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:54:58.324740 1765 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32e172db2844417ea4bd37782e07eeab [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:54:58.324972 1765 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32e172db2844417ea4bd37782e07eeab [term 1 LEADER]: Becoming Leader. State: Replica: 32e172db2844417ea4bd37782e07eeab, State: Running, Role: LEADER
I20260810 03:54:58.325331 1765 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32e172db2844417ea4bd37782e07eeab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e172db2844417ea4bd37782e07eeab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35049 } }
I20260810 03:54:58.325863 1750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32e172db2844417ea4bd37782e07eeab [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:54:58.326902 1766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32e172db2844417ea4bd37782e07eeab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32e172db2844417ea4bd37782e07eeab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e172db2844417ea4bd37782e07eeab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35049 } } }
I20260810 03:54:58.327028 1766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32e172db2844417ea4bd37782e07eeab [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:58.327323 1767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32e172db2844417ea4bd37782e07eeab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32e172db2844417ea4bd37782e07eeab. Latest consensus state: current_term: 1 leader_uuid: "32e172db2844417ea4bd37782e07eeab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e172db2844417ea4bd37782e07eeab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35049 } } }
I20260810 03:54:58.327517 1767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32e172db2844417ea4bd37782e07eeab [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:58.327559 1779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:54:58.330928 1779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:54:58.336941 1779 catalog_manager.cc:1347] Generated new cluster ID: d7bd2db52954464caa94827caee5e346
I20260810 03:54:58.337033 1779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:54:58.350060 1779 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:54:58.350431 1779 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:54:58.361284 1779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32e172db2844417ea4bd37782e07eeab: Generated new TSK 0
I20260810 03:54:58.362290 1779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:54:58.375885 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 1641
2026-08-10T03:54:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1579 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:54:58 dist-test-slave-pww0 krb5kdc[1803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:54:58 dist-test-slave-pww0 krb5kdc[1803](info): set up 2 sockets
Aug 10 03:54:58 dist-test-slave-pww0 krb5kdc[1803](info): commencing operation
krb5kdc: starting...
W20260810 03:54:59.569674 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.170s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:54:59 dist-test-slave-pww0 krb5kdc[1803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:54:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:54:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:54:59.702574 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36575
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:54:59.822942 1829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:54:59.823261 1829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:54:59.823324 1829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:54:59.827440 1829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:54:59.827543 1829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:54:59.827569 1829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:54:59.827590 1829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:54:59.827610 1829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:54:59.833187 1829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36575
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:54:59.834358 1829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:54:59.835438 1829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:54:59.844215 1829 server_base.cc:1034] running on GCE node
W20260810 03:54:59.844183 1841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:59.844120 1838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:54:59.844117 1836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:54:59.845022 1829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:54:59.846199 1829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:54:59.847370 1829 hybrid_clock.cc:648] HybridClock initialized: now 1786334099847355 us; error 29 us; skew 500 ppm
Aug 10 03:54:59 dist-test-slave-pww0 krb5kdc[1803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:54:59.850186 1829 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:54:59.851251 1829 webserver.cc:460] Webserver started at http://127.0.0.1:42727/ using document root <none> and password file <none>
I20260810 03:54:59.851930 1829 fs_manager.cc:359] Metadata directory not provided
I20260810 03:54:59.852025 1829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:54:59.852257 1829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:54:59.854604 1829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "d8c6f0882efb44d69b34afa279ad05eb"
format_stamp: "Formatted at 2026-08-10 03:54:59 on dist-test-slave-pww0"
server_key: "39dcbab838dcc1583d25c678b3e53259"
server_key_iv: "57cef7db11a453f3e5f59e69a8cb9768"
server_key_version: "encryptionkey@0"
I20260810 03:54:59.855244 1829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "d8c6f0882efb44d69b34afa279ad05eb"
format_stamp: "Formatted at 2026-08-10 03:54:59 on dist-test-slave-pww0"
server_key: "39dcbab838dcc1583d25c678b3e53259"
server_key_iv: "57cef7db11a453f3e5f59e69a8cb9768"
server_key_version: "encryptionkey@0"
I20260810 03:54:59.859426 1829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 03:54:59.862141 1852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:54:59.863495 1829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:54:59.863636 1829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "d8c6f0882efb44d69b34afa279ad05eb"
format_stamp: "Formatted at 2026-08-10 03:54:59 on dist-test-slave-pww0"
server_key: "39dcbab838dcc1583d25c678b3e53259"
server_key_iv: "57cef7db11a453f3e5f59e69a8cb9768"
server_key_version: "encryptionkey@0"
I20260810 03:54:59.863749 1829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:54:59.878798 1829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:54:59.882154 1829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:54:59.882366 1829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:54:59.889773 1829 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36575
I20260810 03:54:59.889793 1938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36575 every 8 connection(s)
I20260810 03:54:59.890834 1829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:54:59.893944 1939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:54:59.900189 1939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb: Bootstrap starting.
I20260810 03:54:59.900704 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 1829
I20260810 03:54:59.900853 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:54:59.901156 28261 external_mini_cluster.cc:1428] Setting key 13f6909212f6eb72170fec5299cf1873
I20260810 03:54:59.903606 1939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb: Neither blocks nor log segments found. Creating new log.
I20260810 03:54:59.904417 1939 log.cc:824] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb: Log is configured to *not* fsync() on all Append() calls
I20260810 03:54:59.906693 1939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb: No bootstrap required, opened a new log
Aug 10 03:54:59 dist-test-slave-pww0 krb5kdc[1803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 03:54:59.911365 1939 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c6f0882efb44d69b34afa279ad05eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36575 } }
I20260810 03:54:59.911727 1939 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:54:59.911798 1939 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8c6f0882efb44d69b34afa279ad05eb, State: Initialized, Role: FOLLOWER
I20260810 03:54:59.912297 1939 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c6f0882efb44d69b34afa279ad05eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36575 } }
I20260810 03:54:59.912477 1939 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:54:59.912554 1939 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:54:59.912658 1939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:54:59.913718 1939 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c6f0882efb44d69b34afa279ad05eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36575 } }
I20260810 03:54:59.914129 1939 leader_election.cc:302] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8c6f0882efb44d69b34afa279ad05eb; no voters:
I20260810 03:54:59.914536 1939 leader_election.cc:288] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:54:59.914650 1949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:54:59.914915 1949 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb [term 1 LEADER]: Becoming Leader. State: Replica: d8c6f0882efb44d69b34afa279ad05eb, State: Running, Role: LEADER
I20260810 03:54:59.915939 1939 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:54:59.916064 1949 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c6f0882efb44d69b34afa279ad05eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36575 } }
I20260810 03:54:59.917519 1943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:59.902687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:38411 (local address 127.0.0.1:36575)
0810 03:54:59.903155 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:59.903172 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:59.903213 (+ 41us) server_negotiation.cc:407] Connection header received
0810 03:54:59.904003 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:59.904026 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:59.904424 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:59.904769 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:59.905591 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:59.906621 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:59.907867 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:59.908410 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:59.908552 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:59.911719 (+ 3167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:59.911725 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:59.911742 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:59.911785 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:59.913781 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:59.914431 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:59.914437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:59.914442 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:59.914605 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:59.914996 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:59.914999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:59.915002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:59.915490 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:59.915721 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:59.916215 (+ 494us) server_negotiation.cc:299] Negotiation successful
0810 03:54:59.916485 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":135,"threads_started":1}
I20260810 03:54:59.918002 1955 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8c6f0882efb44d69b34afa279ad05eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c6f0882efb44d69b34afa279ad05eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36575 } } }
I20260810 03:54:59.918169 1955 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:59.918315 1951 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8c6f0882efb44d69b34afa279ad05eb. Latest consensus state: current_term: 1 leader_uuid: "d8c6f0882efb44d69b34afa279ad05eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c6f0882efb44d69b34afa279ad05eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36575 } } }
I20260810 03:54:59.918404 1951 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb [sys.catalog]: This master's current role is: LEADER
I20260810 03:54:59.921366 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260810 03:54:59.921408 1973 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:54:59.921589 1973 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:54:59.921715 1974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:54:59.922672 1974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:54:59.929177 1974 catalog_manager.cc:1347] Generated new cluster ID: 4ce895fed0134c93aa02477f9571645a
I20260810 03:54:59.929252 1974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:54:59.933594 1943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:54:59.923566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:50389 (local address 127.0.0.1:36575)
0810 03:54:59.923682 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 03:54:59.923686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:54:59.925088 (+ 1402us) server_negotiation.cc:407] Connection header received
0810 03:54:59.925212 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:54:59.925215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:54:59.925270 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:54:59.925363 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:54:59.926249 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:59.926957 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:59.927933 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:54:59.928197 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:54:59.928289 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:54:59.930012 (+ 1723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:54:59.930016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:54:59.930019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:54:59.930049 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:54:59.931760 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:59.932298 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:59.932302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:59.932304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:59.932397 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:54:59.932694 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:54:59.932697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:54:59.932699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:54:59.932920 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:54:59.933071 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 03:54:59.933321 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 03:54:59.933425 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 03:54:59.943753 1974 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:54:59.944825 1974 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:54:59.954165 1974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8c6f0882efb44d69b34afa279ad05eb: Generated new TSK 0
I20260810 03:54:59.954991 1974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:54:59.963919 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 1829
W20260810 03:54:59.977792 1801 connection.cc:535] client connection to 127.0.0.1:36575 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36575): Connection reset by peer (error 104)
2026-08-10T03:54:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1593 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:55:00 dist-test-slave-pww0 krb5kdc[2000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:55:00 dist-test-slave-pww0 krb5kdc[2000](info): set up 2 sockets
Aug 10 03:55:00 dist-test-slave-pww0 krb5kdc[2000](info): commencing operation
krb5kdc: starting...
W20260810 03:55:01.164443 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:55:01 dist-test-slave-pww0 krb5kdc[2000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:55:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:55:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:55:01.292630 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43531
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:55:01.410216 2028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:55:01.410534 2028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:55:01.410588 2028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:55:01.414675 2028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:55:01.414767 2028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:55:01.414788 2028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:55:01.414808 2028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:55:01.414824 2028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:55:01.420382 2028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:55:01.421478 2028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:55:01.422482 2028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:55:01.431495 2041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:01.431664 2038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:01.431525 2043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:55:01.431844 2028 server_base.cc:1034] running on GCE node
I20260810 03:55:01.432463 2028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:55:01.433616 2028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:55:01.434799 2028 hybrid_clock.cc:648] HybridClock initialized: now 1786334101434778 us; error 35 us; skew 500 ppm
Aug 10 03:55:01 dist-test-slave-pww0 krb5kdc[2000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:55:01.440791 2028 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:55:01.441857 2028 webserver.cc:460] Webserver started at http://127.0.0.1:45223/ using document root <none> and password file <none>
I20260810 03:55:01.442519 2028 fs_manager.cc:359] Metadata directory not provided
I20260810 03:55:01.442595 2028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:55:01.442811 2028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:55:01.445041 2028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "cf8c79a46a574a03bcc84838676240e4"
format_stamp: "Formatted at 2026-08-10 03:55:01 on dist-test-slave-pww0"
server_key: "14b3ab333cd286cb4578a3649d46eee0"
server_key_iv: "f6dc1bc4a3c416a6dbcce7e2cd22535b"
server_key_version: "encryptionkey@0"
I20260810 03:55:01.445621 2028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "cf8c79a46a574a03bcc84838676240e4"
format_stamp: "Formatted at 2026-08-10 03:55:01 on dist-test-slave-pww0"
server_key: "14b3ab333cd286cb4578a3649d46eee0"
server_key_iv: "f6dc1bc4a3c416a6dbcce7e2cd22535b"
server_key_version: "encryptionkey@0"
I20260810 03:55:01.449625 2028 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:55:01.452179 2057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:55:01.453454 2028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:55:01.453581 2028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "cf8c79a46a574a03bcc84838676240e4"
format_stamp: "Formatted at 2026-08-10 03:55:01 on dist-test-slave-pww0"
server_key: "14b3ab333cd286cb4578a3649d46eee0"
server_key_iv: "f6dc1bc4a3c416a6dbcce7e2cd22535b"
server_key_version: "encryptionkey@0"
I20260810 03:55:01.453687 2028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:55:01.467183 2028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:55:01.470566 2028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:55:01.470769 2028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:55:01.478355 2028 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43531
I20260810 03:55:01.478363 2153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43531 every 8 connection(s)
I20260810 03:55:01.479419 2028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:55:01.480803 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 2028
I20260810 03:55:01.480940 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:55:01.481240 28261 external_mini_cluster.cc:1428] Setting key 3e99811916f8ace16f52894eb76cc4ca
I20260810 03:55:01.483013 2154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:55:01.489781 2154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf8c79a46a574a03bcc84838676240e4: Bootstrap starting.
I20260810 03:55:01.491072 2157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:01.482696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:53075 (local address 127.0.0.1:43531)
0810 03:55:01.483156 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:01.483187 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:01.483227 (+ 40us) server_negotiation.cc:407] Connection header received
0810 03:55:01.483954 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:01.483973 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:01.484353 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:01.484664 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:01.485076 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:01.486130 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:01.487358 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:01.487943 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:01.488075 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:01.489138 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:01.489144 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:01.489160 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:55:01.489576 (+ 416us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:01.489650 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:55:01.489677 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:01.489792 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:01.490069 (+ 277us) server_negotiation.cc:299] Negotiation successful
0810 03:55:01.490245 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":140,"threads_started":1}
I20260810 03:55:01.492457 2154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf8c79a46a574a03bcc84838676240e4: Neither blocks nor log segments found. Creating new log.
I20260810 03:55:01.493356 2154 log.cc:824] T 00000000000000000000000000000000 P cf8c79a46a574a03bcc84838676240e4: Log is configured to *not* fsync() on all Append() calls
I20260810 03:55:01.494522 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:55:01.496006 2154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf8c79a46a574a03bcc84838676240e4: No bootstrap required, opened a new log
I20260810 03:55:01.500041 2154 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf8c79a46a574a03bcc84838676240e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8c79a46a574a03bcc84838676240e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43531 } }
I20260810 03:55:01.500378 2154 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf8c79a46a574a03bcc84838676240e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:55:01.500507 2154 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf8c79a46a574a03bcc84838676240e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf8c79a46a574a03bcc84838676240e4, State: Initialized, Role: FOLLOWER
I20260810 03:55:01.501008 2154 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf8c79a46a574a03bcc84838676240e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8c79a46a574a03bcc84838676240e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43531 } }
I20260810 03:55:01.501168 2154 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf8c79a46a574a03bcc84838676240e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:55:01.501245 2154 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf8c79a46a574a03bcc84838676240e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:55:01.501366 2154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf8c79a46a574a03bcc84838676240e4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:55:01.501816 2157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:01.496988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:48207 (local address 127.0.0.1:43531)
0810 03:55:01.497118 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:01.497122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:01.497273 (+ 151us) server_negotiation.cc:407] Connection header received
0810 03:55:01.497390 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:01.497393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:01.497442 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:01.497518 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:01.497799 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:01.499006 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:01.499945 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:01.500252 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:01.500360 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:01.501252 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:01.501256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:01.501260 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:55:01.501276 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:01.501309 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:55:01.501323 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:01.501420 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:01.501633 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 03:55:01.501678 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 03:55:01.502504 2154 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf8c79a46a574a03bcc84838676240e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8c79a46a574a03bcc84838676240e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43531 } }
I20260810 03:55:01.502862 2154 leader_election.cc:302] T 00000000000000000000000000000000 P cf8c79a46a574a03bcc84838676240e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf8c79a46a574a03bcc84838676240e4; no voters:
I20260810 03:55:01.503183 2154 leader_election.cc:288] T 00000000000000000000000000000000 P cf8c79a46a574a03bcc84838676240e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:55:01.503341 2173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf8c79a46a574a03bcc84838676240e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:55:01.503665 2173 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf8c79a46a574a03bcc84838676240e4 [term 1 LEADER]: Becoming Leader. State: Replica: cf8c79a46a574a03bcc84838676240e4, State: Running, Role: LEADER
I20260810 03:55:01.504009 2173 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf8c79a46a574a03bcc84838676240e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8c79a46a574a03bcc84838676240e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43531 } }
I20260810 03:55:01.504513 2154 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf8c79a46a574a03bcc84838676240e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:55:01.505843 2179 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf8c79a46a574a03bcc84838676240e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf8c79a46a574a03bcc84838676240e4. Latest consensus state: current_term: 1 leader_uuid: "cf8c79a46a574a03bcc84838676240e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8c79a46a574a03bcc84838676240e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43531 } } }
I20260810 03:55:01.506053 2179 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf8c79a46a574a03bcc84838676240e4 [sys.catalog]: This master's current role is: LEADER
I20260810 03:55:01.505815 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf8c79a46a574a03bcc84838676240e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf8c79a46a574a03bcc84838676240e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8c79a46a574a03bcc84838676240e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43531 } } }
I20260810 03:55:01.506383 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf8c79a46a574a03bcc84838676240e4 [sys.catalog]: This master's current role is: LEADER
I20260810 03:55:01.506520 2202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:55:01.509449 2202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:55:01.515218 2202 catalog_manager.cc:1347] Generated new cluster ID: 019b4aaf15c04ecfa456348f0c8d24cc
I20260810 03:55:01.515311 2202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:55:01.536204 2202 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:55:01.537606 2202 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:55:01.546973 2202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf8c79a46a574a03bcc84838676240e4: Generated new TSK 0
I20260810 03:55:01.547976 2202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:55:01.554848 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 2028
W20260810 03:55:01.572513 1998 connection.cc:535] client connection to 127.0.0.1:43531 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43531): Connection reset by peer (error 104)
2026-08-10T03:55:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1594 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:55:01 dist-test-slave-pww0 krb5kdc[2212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:55:01 dist-test-slave-pww0 krb5kdc[2212](info): set up 2 sockets
Aug 10 03:55:01 dist-test-slave-pww0 krb5kdc[2212](info): commencing operation
krb5kdc: starting...
W20260810 03:55:02.765689 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.179s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:55:02 dist-test-slave-pww0 krb5kdc[2212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:55:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:55:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 03:55:02.896847 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34883
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:55:03.014562 2241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:55:03.014880 2241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:55:03.014926 2241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:55:03.019192 2241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:55:03.019299 2241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:55:03.019323 2241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:55:03.019343 2241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:55:03.019361 2241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:55:03.025305 2241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34883
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:55:03.026463 2241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:55:03.027552 2241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:55:03.036587 2257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:03.036602 2253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:03.036643 2251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:55:03.036758 2241 server_base.cc:1034] running on GCE node
I20260810 03:55:03.037465 2241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:55:03.038650 2241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:55:03.039855 2241 hybrid_clock.cc:648] HybridClock initialized: now 1786334103039823 us; error 48 us; skew 500 ppm
Aug 10 03:55:03 dist-test-slave-pww0 krb5kdc[2212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:55:03.042743 2241 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 03:55:03.043833 2241 webserver.cc:460] Webserver started at http://127.0.0.1:39639/ using document root <none> and password file <none>
I20260810 03:55:03.044413 2241 fs_manager.cc:359] Metadata directory not provided
I20260810 03:55:03.044476 2241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:55:03.044656 2241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:55:03.046844 2241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "473d980782f3413eba314b00ba79dfa7"
format_stamp: "Formatted at 2026-08-10 03:55:03 on dist-test-slave-pww0"
server_key: "42adb906dd82b556ea09766c03855463"
server_key_iv: "bb3c9cfd8bd0136e52f523ea6117ac97"
server_key_version: "encryptionkey@0"
I20260810 03:55:03.047420 2241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "473d980782f3413eba314b00ba79dfa7"
format_stamp: "Formatted at 2026-08-10 03:55:03 on dist-test-slave-pww0"
server_key: "42adb906dd82b556ea09766c03855463"
server_key_iv: "bb3c9cfd8bd0136e52f523ea6117ac97"
server_key_version: "encryptionkey@0"
I20260810 03:55:03.051404 2241 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:55:03.054011 2268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:55:03.055335 2241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 03:55:03.055514 2241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "473d980782f3413eba314b00ba79dfa7"
format_stamp: "Formatted at 2026-08-10 03:55:03 on dist-test-slave-pww0"
server_key: "42adb906dd82b556ea09766c03855463"
server_key_iv: "bb3c9cfd8bd0136e52f523ea6117ac97"
server_key_version: "encryptionkey@0"
I20260810 03:55:03.055632 2241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:55:03.060412 2241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:55:03.063611 2241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:55:03.063781 2241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:55:03.071376 2241 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34883
I20260810 03:55:03.071391 2367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34883 every 8 connection(s)
I20260810 03:55:03.072407 2241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:55:03.074730 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 2241
I20260810 03:55:03.074868 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:55:03.075167 28261 external_mini_cluster.cc:1428] Setting key 6887932cf7a89f7cc0235c4629af7e49
I20260810 03:55:03.075608 2375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:55:03.080753 2379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:03.076715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:47615 (local address 127.0.0.1:34883)
0810 03:55:03.077143 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:03.077159 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:03.077197 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:55:03.077868 (+ 671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:03.077884 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:03.078178 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:03.078469 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:03.079436 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:03.079442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:03.079483 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:55:03.079501 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:03.079568 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:55:03.079591 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:03.079714 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:03.079835 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 03:55:03.079997 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":127,"threads_started":1}
I20260810 03:55:03.082813 2375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 473d980782f3413eba314b00ba79dfa7: Bootstrap starting.
I20260810 03:55:03.083798 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:55:03.085759 2375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 473d980782f3413eba314b00ba79dfa7: Neither blocks nor log segments found. Creating new log.
I20260810 03:55:03.086757 2375 log.cc:824] T 00000000000000000000000000000000 P 473d980782f3413eba314b00ba79dfa7: Log is configured to *not* fsync() on all Append() calls
I20260810 03:55:03.087879 2379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:03.086168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.195:47953 (local address 127.0.0.1:34883)
0810 03:55:03.086296 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:03.086300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:03.086450 (+ 150us) server_negotiation.cc:407] Connection header received
0810 03:55:03.086548 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:03.086551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:03.086592 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:03.086682 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:03.087432 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:03.087436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:03.087438 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:55:03.087448 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:03.087510 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:55:03.087522 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:03.087592 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:03.087680 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 03:55:03.087724 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 03:55:03.089187 2375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 473d980782f3413eba314b00ba79dfa7: No bootstrap required, opened a new log
I20260810 03:55:03.093036 2375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 473d980782f3413eba314b00ba79dfa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473d980782f3413eba314b00ba79dfa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34883 } }
I20260810 03:55:03.093318 2375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 473d980782f3413eba314b00ba79dfa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:55:03.093357 2375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 473d980782f3413eba314b00ba79dfa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 473d980782f3413eba314b00ba79dfa7, State: Initialized, Role: FOLLOWER
I20260810 03:55:03.093828 2375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 473d980782f3413eba314b00ba79dfa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473d980782f3413eba314b00ba79dfa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34883 } }
I20260810 03:55:03.093961 2375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 473d980782f3413eba314b00ba79dfa7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:55:03.094025 2375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 473d980782f3413eba314b00ba79dfa7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:55:03.094125 2375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 473d980782f3413eba314b00ba79dfa7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:55:03.095139 2375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 473d980782f3413eba314b00ba79dfa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473d980782f3413eba314b00ba79dfa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34883 } }
I20260810 03:55:03.095552 2375 leader_election.cc:302] T 00000000000000000000000000000000 P 473d980782f3413eba314b00ba79dfa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 473d980782f3413eba314b00ba79dfa7; no voters:
I20260810 03:55:03.095892 2375 leader_election.cc:288] T 00000000000000000000000000000000 P 473d980782f3413eba314b00ba79dfa7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:55:03.096002 2395 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 473d980782f3413eba314b00ba79dfa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:55:03.096274 2395 raft_consensus.cc:686] T 00000000000000000000000000000000 P 473d980782f3413eba314b00ba79dfa7 [term 1 LEADER]: Becoming Leader. State: Replica: 473d980782f3413eba314b00ba79dfa7, State: Running, Role: LEADER
I20260810 03:55:03.096644 2395 consensus_queue.cc:237] T 00000000000000000000000000000000 P 473d980782f3413eba314b00ba79dfa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473d980782f3413eba314b00ba79dfa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34883 } }
I20260810 03:55:03.097244 2375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 473d980782f3413eba314b00ba79dfa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:55:03.098328 2400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 473d980782f3413eba314b00ba79dfa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 473d980782f3413eba314b00ba79dfa7. Latest consensus state: current_term: 1 leader_uuid: "473d980782f3413eba314b00ba79dfa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473d980782f3413eba314b00ba79dfa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34883 } } }
I20260810 03:55:03.098477 2400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 473d980782f3413eba314b00ba79dfa7 [sys.catalog]: This master's current role is: LEADER
I20260810 03:55:03.098666 2396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 473d980782f3413eba314b00ba79dfa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "473d980782f3413eba314b00ba79dfa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "473d980782f3413eba314b00ba79dfa7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34883 } } }
I20260810 03:55:03.098865 2396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 473d980782f3413eba314b00ba79dfa7 [sys.catalog]: This master's current role is: LEADER
I20260810 03:55:03.099175 2414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:55:03.101974 2414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:55:03.107417 2414 catalog_manager.cc:1347] Generated new cluster ID: b36ae58abe4d4583989f47ae14887cc6
I20260810 03:55:03.107581 2414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:55:03.135860 2414 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:55:03.136253 2414 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:55:03.148388 2414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 473d980782f3413eba314b00ba79dfa7: Generated new TSK 0
I20260810 03:55:03.149348 2414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:55:03.152681 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 2241
2026-08-10T03:55:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1603 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:55:03 dist-test-slave-pww0 krb5kdc[2435](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:55:03 dist-test-slave-pww0 krb5kdc[2435](info): set up 2 sockets
Aug 10 03:55:03 dist-test-slave-pww0 krb5kdc[2435](info): commencing operation
krb5kdc: starting...
W20260810 03:55:04.368091 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:55:04 dist-test-slave-pww0 krb5kdc[2435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:55:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:55:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:55:04.504117 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:37441
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:43611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:37441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:55:04.620042 2463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:55:04.620338 2463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:55:04.620388 2463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:55:04.624495 2463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:55:04.624574 2463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:55:04.624593 2463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:55:04.624609 2463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:55:04.624624 2463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:55:04.630239 2463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:43611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:37441
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:37441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:55:04.631556 2463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:55:04.632680 2463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:55:04.641774 2475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:55:04.641950 2463 server_base.cc:1034] running on GCE node
W20260810 03:55:04.641748 2473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:04.641744 2480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:55:04.642747 2463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:55:04.644083 2463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:55:04.645257 2463 hybrid_clock.cc:648] HybridClock initialized: now 1786334104645245 us; error 41 us; skew 500 ppm
Aug 10 03:55:04 dist-test-slave-pww0 krb5kdc[2435](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334104, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:55:04.648322 2463 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:55:04.649381 2463 webserver.cc:460] Webserver started at http://127.27.153.126:41265/ using document root <none> and password file <none>
I20260810 03:55:04.649986 2463 fs_manager.cc:359] Metadata directory not provided
I20260810 03:55:04.650063 2463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:55:04.650303 2463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:55:04.652666 2463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "64b4c9ab7bb04e8fb72c71df35094740"
format_stamp: "Formatted at 2026-08-10 03:55:04 on dist-test-slave-pww0"
server_key: "37af1b3af13b2141bb2a390896dbd951"
server_key_iv: "62700ed67c84dd1c3c8992261420dd71"
server_key_version: "encryptionkey@0"
I20260810 03:55:04.653291 2463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "64b4c9ab7bb04e8fb72c71df35094740"
format_stamp: "Formatted at 2026-08-10 03:55:04 on dist-test-slave-pww0"
server_key: "37af1b3af13b2141bb2a390896dbd951"
server_key_iv: "62700ed67c84dd1c3c8992261420dd71"
server_key_version: "encryptionkey@0"
I20260810 03:55:04.657337 2463 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:55:04.659933 2491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:55:04.661252 2463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 03:55:04.661406 2463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "64b4c9ab7bb04e8fb72c71df35094740"
format_stamp: "Formatted at 2026-08-10 03:55:04 on dist-test-slave-pww0"
server_key: "37af1b3af13b2141bb2a390896dbd951"
server_key_iv: "62700ed67c84dd1c3c8992261420dd71"
server_key_version: "encryptionkey@0"
I20260810 03:55:04.661520 2463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:55:04.679414 2463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:55:04.682865 2463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:55:04.683068 2463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:55:04.690606 2463 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:37441
I20260810 03:55:04.690615 2570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:37441 every 8 connection(s)
I20260810 03:55:04.691733 2463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:55:04.691998 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 2463
I20260810 03:55:04.692119 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:55:04.692442 28261 external_mini_cluster.cc:1428] Setting key 1d853110db110b6b91001322bcf1f37b
I20260810 03:55:04.695034 2572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:55:04.701149 2572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64b4c9ab7bb04e8fb72c71df35094740: Bootstrap starting.
Aug 10 03:55:04 dist-test-slave-pww0 krb5kdc[2435](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:55:04.703687 2572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64b4c9ab7bb04e8fb72c71df35094740: Neither blocks nor log segments found. Creating new log.
I20260810 03:55:04.704453 2572 log.cc:824] T 00000000000000000000000000000000 P 64b4c9ab7bb04e8fb72c71df35094740: Log is configured to *not* fsync() on all Append() calls
I20260810 03:55:04.706542 2572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64b4c9ab7bb04e8fb72c71df35094740: No bootstrap required, opened a new log
I20260810 03:55:04.706564 2581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:04.693858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38189 (local address 127.27.153.126:37441)
0810 03:55:04.694354 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:04.694371 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:04.694409 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:55:04.695062 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:04.695078 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:04.695486 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:04.695820 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:04.696657 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:04.697754 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:04.698635 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:04.699113 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:04.699259 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:04.701845 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:04.701851 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:04.701865 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:04.701897 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:04.703599 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:04.704149 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:04.704154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:04.704159 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:04.704280 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:04.704624 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:04.704627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:04.704628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:04.705050 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:04.705252 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:04.705599 (+ 347us) server_negotiation.cc:299] Negotiation successful
0810 03:55:04.705830 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":165,"threads_started":1}
I20260810 03:55:04.709613 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:55:04.710577 2572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64b4c9ab7bb04e8fb72c71df35094740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b4c9ab7bb04e8fb72c71df35094740" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37441 } }
I20260810 03:55:04.710889 2572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64b4c9ab7bb04e8fb72c71df35094740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:55:04.710996 2572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64b4c9ab7bb04e8fb72c71df35094740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b4c9ab7bb04e8fb72c71df35094740, State: Initialized, Role: FOLLOWER
I20260810 03:55:04.711503 2572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64b4c9ab7bb04e8fb72c71df35094740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b4c9ab7bb04e8fb72c71df35094740" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37441 } }
I20260810 03:55:04.711668 2572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64b4c9ab7bb04e8fb72c71df35094740 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:55:04.711741 2572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64b4c9ab7bb04e8fb72c71df35094740 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:55:04.711856 2572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64b4c9ab7bb04e8fb72c71df35094740 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:55:04.713024 2572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64b4c9ab7bb04e8fb72c71df35094740 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b4c9ab7bb04e8fb72c71df35094740" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37441 } }
I20260810 03:55:04.713430 2572 leader_election.cc:302] T 00000000000000000000000000000000 P 64b4c9ab7bb04e8fb72c71df35094740 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64b4c9ab7bb04e8fb72c71df35094740; no voters:
I20260810 03:55:04.713800 2572 leader_election.cc:288] T 00000000000000000000000000000000 P 64b4c9ab7bb04e8fb72c71df35094740 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:55:04.713900 2590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64b4c9ab7bb04e8fb72c71df35094740 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:55:04.714136 2590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64b4c9ab7bb04e8fb72c71df35094740 [term 1 LEADER]: Becoming Leader. State: Replica: 64b4c9ab7bb04e8fb72c71df35094740, State: Running, Role: LEADER
I20260810 03:55:04.714540 2590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64b4c9ab7bb04e8fb72c71df35094740 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b4c9ab7bb04e8fb72c71df35094740" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37441 } }
I20260810 03:55:04.715233 2572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64b4c9ab7bb04e8fb72c71df35094740 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:55:04.716320 2592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64b4c9ab7bb04e8fb72c71df35094740 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64b4c9ab7bb04e8fb72c71df35094740" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b4c9ab7bb04e8fb72c71df35094740" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37441 } } }
I20260810 03:55:04.716460 2592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64b4c9ab7bb04e8fb72c71df35094740 [sys.catalog]: This master's current role is: LEADER
I20260810 03:55:04.716856 2608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:55:04.717406 2594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64b4c9ab7bb04e8fb72c71df35094740 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64b4c9ab7bb04e8fb72c71df35094740. Latest consensus state: current_term: 1 leader_uuid: "64b4c9ab7bb04e8fb72c71df35094740" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b4c9ab7bb04e8fb72c71df35094740" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 37441 } } }
I20260810 03:55:04.717579 2594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64b4c9ab7bb04e8fb72c71df35094740 [sys.catalog]: This master's current role is: LEADER
I20260810 03:55:04.720363 2608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:55:04.722643 2581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:04.711620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37587 (local address 127.27.153.126:37441)
0810 03:55:04.711757 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:04.711762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:04.711972 (+ 210us) server_negotiation.cc:407] Connection header received
0810 03:55:04.712143 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:04.712147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:04.712199 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:04.712295 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:04.712922 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:04.713674 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:04.714782 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:04.715122 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:04.715229 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:04.718368 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:04.718373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:04.718376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:04.718408 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:04.720150 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:04.721112 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:04.721116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:04.721118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:04.721215 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:04.721630 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:04.721634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:04.721636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:04.721886 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:04.722053 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:04.722371 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 03:55:04.722483 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 03:55:04.727825 2608 catalog_manager.cc:1347] Generated new cluster ID: 2f0d0a02800f4da18db986b998cfc97e
I20260810 03:55:04.727921 2608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:55:04.745376 2608 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:55:04.746462 2608 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:55:04.754848 2608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64b4c9ab7bb04e8fb72c71df35094740: Generated new TSK 0
I20260810 03:55:04.755669 2608 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:55:04.761453 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 2463
W20260810 03:55:04.770424 2432 connection.cc:535] client connection to 127.27.153.126:37441 recv error: Network error: failed to read from TLS socket (remote: 127.27.153.126:37441): Connection reset by peer (error 104)
2026-08-10T03:55:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1593 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:55:04 dist-test-slave-pww0 krb5kdc[2630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:55:04 dist-test-slave-pww0 krb5kdc[2630](info): set up 2 sockets
Aug 10 03:55:04 dist-test-slave-pww0 krb5kdc[2630](info): commencing operation
krb5kdc: starting...
W20260810 03:55:05.961516 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:55:06 dist-test-slave-pww0 krb5kdc[2630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:55:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:55:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:55:06.095268 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:42605
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:43219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:42605
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:55:06.215785 2655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:55:06.216121 2655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:55:06.216192 2655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:55:06.220418 2655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:55:06.220500 2655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:55:06.220527 2655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:55:06.220551 2655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:55:06.220573 2655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:55:06.226275 2655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:43219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:42605
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:42605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:55:06.227516 2655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:55:06.228637 2655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:55:06.238026 2655 server_base.cc:1034] running on GCE node
W20260810 03:55:06.237922 2668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:06.237922 2665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:06.237928 2673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:55:06.238843 2655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:55:06.240144 2655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:55:06.241334 2655 hybrid_clock.cc:648] HybridClock initialized: now 1786334106241303 us; error 42 us; skew 500 ppm
Aug 10 03:55:06 dist-test-slave-pww0 krb5kdc[2630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334106, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:55:06.244508 2655 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:55:06.245558 2655 webserver.cc:460] Webserver started at http://127.27.153.126:43697/ using document root <none> and password file <none>
I20260810 03:55:06.246167 2655 fs_manager.cc:359] Metadata directory not provided
I20260810 03:55:06.246237 2655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:55:06.246447 2655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:55:06.248765 2655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "f5fad2817e0745f8996adb21c0c4903b"
format_stamp: "Formatted at 2026-08-10 03:55:06 on dist-test-slave-pww0"
server_key: "c4a609dba4d80abb01aa76314d866ad3"
server_key_iv: "a07682b3823bc826fa05af3a1aa713a6"
server_key_version: "encryptionkey@0"
I20260810 03:55:06.249383 2655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "f5fad2817e0745f8996adb21c0c4903b"
format_stamp: "Formatted at 2026-08-10 03:55:06 on dist-test-slave-pww0"
server_key: "c4a609dba4d80abb01aa76314d866ad3"
server_key_iv: "a07682b3823bc826fa05af3a1aa713a6"
server_key_version: "encryptionkey@0"
I20260810 03:55:06.253695 2655 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 03:55:06.256398 2682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:55:06.257851 2655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:55:06.258003 2655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "f5fad2817e0745f8996adb21c0c4903b"
format_stamp: "Formatted at 2026-08-10 03:55:06 on dist-test-slave-pww0"
server_key: "c4a609dba4d80abb01aa76314d866ad3"
server_key_iv: "a07682b3823bc826fa05af3a1aa713a6"
server_key_version: "encryptionkey@0"
I20260810 03:55:06.258122 2655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:55:06.286994 2655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:55:06.290524 2655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:55:06.290740 2655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:55:06.298219 2655 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:42605
I20260810 03:55:06.298249 2781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:42605 every 8 connection(s)
I20260810 03:55:06.299322 2655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:55:06.302421 2784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:55:06.303797 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 2655
I20260810 03:55:06.303946 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:55:06.304260 28261 external_mini_cluster.cc:1428] Setting key ee8c23f18ef220912b805c1b67ac40f9
I20260810 03:55:06.309197 2784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5fad2817e0745f8996adb21c0c4903b: Bootstrap starting.
I20260810 03:55:06.311710 2784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5fad2817e0745f8996adb21c0c4903b: Neither blocks nor log segments found. Creating new log.
I20260810 03:55:06.312549 2784 log.cc:824] T 00000000000000000000000000000000 P f5fad2817e0745f8996adb21c0c4903b: Log is configured to *not* fsync() on all Append() calls
I20260810 03:55:06.313932 2788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:06.305940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53275 (local address 127.27.153.126:42605)
0810 03:55:06.306456 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:06.306478 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:06.306516 (+ 38us) server_negotiation.cc:407] Connection header received
0810 03:55:06.307200 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:06.307219 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:06.307659 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:06.307986 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:06.308444 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:06.309462 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:06.310385 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:06.310878 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:06.311027 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:06.312140 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:06.312145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:06.312159 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:55:06.312588 (+ 429us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:06.312650 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:55:06.312674 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:06.312838 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:06.312987 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 03:55:06.313141 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":145,"threads_started":1}
I20260810 03:55:06.314688 2784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5fad2817e0745f8996adb21c0c4903b: No bootstrap required, opened a new log
I20260810 03:55:06.316882 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:55:06.318645 2784 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5fad2817e0745f8996adb21c0c4903b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5fad2817e0745f8996adb21c0c4903b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42605 } }
I20260810 03:55:06.318974 2784 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5fad2817e0745f8996adb21c0c4903b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:55:06.319089 2784 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5fad2817e0745f8996adb21c0c4903b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5fad2817e0745f8996adb21c0c4903b, State: Initialized, Role: FOLLOWER
I20260810 03:55:06.319635 2784 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5fad2817e0745f8996adb21c0c4903b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5fad2817e0745f8996adb21c0c4903b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42605 } }
I20260810 03:55:06.319820 2784 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5fad2817e0745f8996adb21c0c4903b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:55:06.319900 2784 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5fad2817e0745f8996adb21c0c4903b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:55:06.320005 2784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5fad2817e0745f8996adb21c0c4903b [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:55:06.321004 2784 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5fad2817e0745f8996adb21c0c4903b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5fad2817e0745f8996adb21c0c4903b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42605 } }
I20260810 03:55:06.321367 2784 leader_election.cc:302] T 00000000000000000000000000000000 P f5fad2817e0745f8996adb21c0c4903b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5fad2817e0745f8996adb21c0c4903b; no voters:
I20260810 03:55:06.321693 2784 leader_election.cc:288] T 00000000000000000000000000000000 P f5fad2817e0745f8996adb21c0c4903b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:55:06.321782 2803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5fad2817e0745f8996adb21c0c4903b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:55:06.322001 2803 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5fad2817e0745f8996adb21c0c4903b [term 1 LEADER]: Becoming Leader. State: Replica: f5fad2817e0745f8996adb21c0c4903b, State: Running, Role: LEADER
I20260810 03:55:06.322322 2803 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5fad2817e0745f8996adb21c0c4903b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5fad2817e0745f8996adb21c0c4903b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42605 } }
I20260810 03:55:06.323177 2784 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5fad2817e0745f8996adb21c0c4903b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:55:06.323221 2788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:06.318974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36999 (local address 127.27.153.126:42605)
0810 03:55:06.319103 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:06.319108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:06.319431 (+ 323us) server_negotiation.cc:407] Connection header received
0810 03:55:06.319591 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:06.319595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:06.319657 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:06.319745 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:06.320015 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:06.320586 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:06.321473 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:06.321776 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:06.321875 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:06.322771 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:06.322774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:06.322777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:55:06.322791 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:06.322818 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:55:06.322829 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:06.322899 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:06.323041 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 03:55:06.323082 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 03:55:06.323732 2803 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5fad2817e0745f8996adb21c0c4903b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5fad2817e0745f8996adb21c0c4903b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5fad2817e0745f8996adb21c0c4903b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42605 } } }
I20260810 03:55:06.323920 2803 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5fad2817e0745f8996adb21c0c4903b [sys.catalog]: This master's current role is: LEADER
I20260810 03:55:06.324177 2803 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5fad2817e0745f8996adb21c0c4903b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5fad2817e0745f8996adb21c0c4903b. Latest consensus state: current_term: 1 leader_uuid: "f5fad2817e0745f8996adb21c0c4903b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5fad2817e0745f8996adb21c0c4903b" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42605 } } }
I20260810 03:55:06.324326 2803 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5fad2817e0745f8996adb21c0c4903b [sys.catalog]: This master's current role is: LEADER
I20260810 03:55:06.324738 2819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:55:06.328109 2819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:55:06.333775 2819 catalog_manager.cc:1347] Generated new cluster ID: 1b30fc45457a4b9299e387965ec6c802
I20260810 03:55:06.333874 2819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:55:06.364559 2819 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:55:06.365582 2819 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:55:06.374315 2819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5fad2817e0745f8996adb21c0c4903b: Generated new TSK 0
I20260810 03:55:06.375145 2819 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:55:06.379374 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 2655
W20260810 03:55:06.405144 2625 connection.cc:535] client connection to 127.27.153.126:42605 recv error: Network error: failed to read from TLS socket (remote: 127.27.153.126:42605): Connection reset by peer (error 104)
2026-08-10T03:55:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1634 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:55:06 dist-test-slave-pww0 krb5kdc[2852](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:55:06 dist-test-slave-pww0 krb5kdc[2852](info): set up 2 sockets
Aug 10 03:55:06 dist-test-slave-pww0 krb5kdc[2852](info): commencing operation
krb5kdc: starting...
W20260810 03:55:07.600359 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.181s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:55:07 dist-test-slave-pww0 krb5kdc[2852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:55:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:55:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:55:07.738672 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:43729
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:33277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:43729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:55:07.858081 2873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:55:07.858408 2873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:55:07.858474 2873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:55:07.862718 2873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:55:07.862811 2873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:55:07.862840 2873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:55:07.862861 2873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:55:07.862882 2873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:55:07.869092 2873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:33277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:43729
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:43729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:55:07.870347 2873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:55:07.871480 2873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:55:07.880366 2881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:07.880378 2889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:55:07.880507 2873 server_base.cc:1034] running on GCE node
W20260810 03:55:07.880431 2883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:55:07.881234 2873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:55:07.882464 2873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:55:07.883632 2873 hybrid_clock.cc:648] HybridClock initialized: now 1786334107883616 us; error 29 us; skew 500 ppm
Aug 10 03:55:07 dist-test-slave-pww0 krb5kdc[2852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334107, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:55:07.886713 2873 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:55:07.887826 2873 webserver.cc:460] Webserver started at http://127.27.153.126:45679/ using document root <none> and password file <none>
I20260810 03:55:07.888412 2873 fs_manager.cc:359] Metadata directory not provided
I20260810 03:55:07.888489 2873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:55:07.888715 2873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:55:07.890967 2873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "b0dfc9a953a049c68cede39b965f73b5"
format_stamp: "Formatted at 2026-08-10 03:55:07 on dist-test-slave-pww0"
server_key: "4e76fa47968385e56b62ff22037713c5"
server_key_iv: "6cebfd08d9915bf9e75bd356853664ab"
server_key_version: "encryptionkey@0"
I20260810 03:55:07.891605 2873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "b0dfc9a953a049c68cede39b965f73b5"
format_stamp: "Formatted at 2026-08-10 03:55:07 on dist-test-slave-pww0"
server_key: "4e76fa47968385e56b62ff22037713c5"
server_key_iv: "6cebfd08d9915bf9e75bd356853664ab"
server_key_version: "encryptionkey@0"
I20260810 03:55:07.895699 2873 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:55:07.898427 2902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:55:07.899807 2873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:55:07.899940 2873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "b0dfc9a953a049c68cede39b965f73b5"
format_stamp: "Formatted at 2026-08-10 03:55:07 on dist-test-slave-pww0"
server_key: "4e76fa47968385e56b62ff22037713c5"
server_key_iv: "6cebfd08d9915bf9e75bd356853664ab"
server_key_version: "encryptionkey@0"
I20260810 03:55:07.900056 2873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:55:07.904876 2873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:55:07.908288 2873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:55:07.908483 2873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:55:07.915797 2873 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:43729
I20260810 03:55:07.915832 2985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:43729 every 8 connection(s)
I20260810 03:55:07.916823 2873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:55:07.919823 2986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:55:07.925734 2986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5: Bootstrap starting.
I20260810 03:55:07.927037 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 2873
I20260810 03:55:07.927160 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:55:07.927440 28261 external_mini_cluster.cc:1428] Setting key 645cd06dbca9afcf4148d508295d39ef
I20260810 03:55:07.928709 2986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5: Neither blocks nor log segments found. Creating new log.
I20260810 03:55:07.929802 2986 log.cc:824] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5: Log is configured to *not* fsync() on all Append() calls
I20260810 03:55:07.932413 2986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5: No bootstrap required, opened a new log
I20260810 03:55:07.933476 2992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:07.928712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41873 (local address 127.27.153.126:43729)
0810 03:55:07.929339 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:07.929358 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:07.929406 (+ 48us) server_negotiation.cc:407] Connection header received
0810 03:55:07.930357 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:07.930380 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:07.930700 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:07.931112 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:07.931977 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:07.931983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:07.931997 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:55:07.932010 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:07.932082 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:55:07.932110 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:07.932234 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:07.932306 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 03:55:07.932480 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":206,"threads_started":1}
I20260810 03:55:07.936635 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:55:07.936890 2986 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0dfc9a953a049c68cede39b965f73b5" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 43729 } }
I20260810 03:55:07.937143 2986 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:55:07.937197 2986 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0dfc9a953a049c68cede39b965f73b5, State: Initialized, Role: FOLLOWER
I20260810 03:55:07.937638 2986 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0dfc9a953a049c68cede39b965f73b5" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 43729 } }
I20260810 03:55:07.937770 2986 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:55:07.937821 2986 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:55:07.937918 2986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:55:07.939177 2986 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0dfc9a953a049c68cede39b965f73b5" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 43729 } }
I20260810 03:55:07.939615 2986 leader_election.cc:302] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0dfc9a953a049c68cede39b965f73b5; no voters:
I20260810 03:55:07.939941 2986 leader_election.cc:288] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:55:07.940044 3008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:55:07.940271 3008 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5 [term 1 LEADER]: Becoming Leader. State: Replica: b0dfc9a953a049c68cede39b965f73b5, State: Running, Role: LEADER
I20260810 03:55:07.940656 3008 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0dfc9a953a049c68cede39b965f73b5" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 43729 } }
I20260810 03:55:07.940800 2992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:07.938878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45789 (local address 127.27.153.126:43729)
0810 03:55:07.939002 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:07.939006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:07.939190 (+ 184us) server_negotiation.cc:407] Connection header received
0810 03:55:07.939340 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:07.939342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:07.939382 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:07.939452 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:07.940302 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:07.940306 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:07.940309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:55:07.940321 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:07.940353 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:55:07.940367 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:07.940427 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:07.940601 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 03:55:07.940647 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 03:55:07.941481 2986 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:55:07.942525 3008 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0dfc9a953a049c68cede39b965f73b5. Latest consensus state: current_term: 1 leader_uuid: "b0dfc9a953a049c68cede39b965f73b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0dfc9a953a049c68cede39b965f73b5" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 43729 } } }
I20260810 03:55:07.942672 3008 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5 [sys.catalog]: This master's current role is: LEADER
I20260810 03:55:07.943240 3009 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0dfc9a953a049c68cede39b965f73b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0dfc9a953a049c68cede39b965f73b5" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 43729 } } }
I20260810 03:55:07.943897 3009 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5 [sys.catalog]: This master's current role is: LEADER
W20260810 03:55:07.947490 3032 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:55:07.947597 3032 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:55:07.947664 3029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:55:07.948395 3029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:55:07.954054 3029 catalog_manager.cc:1347] Generated new cluster ID: 23a9d9ff3ce44823a00d556eadf8a00b
I20260810 03:55:07.954154 3029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:55:07.966377 3029 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:55:07.966775 3029 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:55:07.979272 3029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0dfc9a953a049c68cede39b965f73b5: Generated new TSK 0
I20260810 03:55:07.980304 3029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:55:07.984644 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 2873
2026-08-10T03:55:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1621 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:55:08 dist-test-slave-pww0 krb5kdc[3041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:55:08 dist-test-slave-pww0 krb5kdc[3041](info): set up 2 sockets
Aug 10 03:55:08 dist-test-slave-pww0 krb5kdc[3041](info): commencing operation
krb5kdc: starting...
W20260810 03:55:09.213838 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:55:09 dist-test-slave-pww0 krb5kdc[3041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:55:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:55:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:55:09.343590 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:40603
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:37705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:40603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:55:09.460500 3061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:55:09.460815 3061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:55:09.460880 3061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:55:09.464988 3061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:55:09.465068 3061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:55:09.465099 3061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:55:09.465121 3061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:55:09.465143 3061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:55:09.470731 3061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:37705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:40603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:40603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:55:09.471941 3061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:55:09.473008 3061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 03:55:09.481884 3061 server_base.cc:1034] running on GCE node
W20260810 03:55:09.481840 3066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:09.481911 3067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:09.481840 3071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:55:09.482760 3061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:55:09.484017 3061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:55:09.485221 3061 hybrid_clock.cc:648] HybridClock initialized: now 1786334109485196 us; error 37 us; skew 500 ppm
Aug 10 03:55:09 dist-test-slave-pww0 krb5kdc[3041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334109, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:55:09.488202 3061 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:55:09.489166 3061 webserver.cc:460] Webserver started at http://127.27.153.126:38857/ using document root <none> and password file <none>
I20260810 03:55:09.489717 3061 fs_manager.cc:359] Metadata directory not provided
I20260810 03:55:09.489818 3061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:55:09.490038 3061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:55:09.492290 3061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "5ba07a0129e646aaae5cfcec225d5d19"
format_stamp: "Formatted at 2026-08-10 03:55:09 on dist-test-slave-pww0"
server_key: "d3428274a61ffc57da0e2270223445a5"
server_key_iv: "1c4e62238c1ce9087b2d8bc60765c1c2"
server_key_version: "encryptionkey@0"
I20260810 03:55:09.492875 3061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "5ba07a0129e646aaae5cfcec225d5d19"
format_stamp: "Formatted at 2026-08-10 03:55:09 on dist-test-slave-pww0"
server_key: "d3428274a61ffc57da0e2270223445a5"
server_key_iv: "1c4e62238c1ce9087b2d8bc60765c1c2"
server_key_version: "encryptionkey@0"
I20260810 03:55:09.496852 3061 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:55:09.499437 3083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:55:09.500808 3061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:55:09.500933 3061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "5ba07a0129e646aaae5cfcec225d5d19"
format_stamp: "Formatted at 2026-08-10 03:55:09 on dist-test-slave-pww0"
server_key: "d3428274a61ffc57da0e2270223445a5"
server_key_iv: "1c4e62238c1ce9087b2d8bc60765c1c2"
server_key_version: "encryptionkey@0"
I20260810 03:55:09.501039 3061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:55:09.529290 3061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:55:09.532405 3061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:55:09.532596 3061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:55:09.539639 3061 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:40603
I20260810 03:55:09.539664 3177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:40603 every 8 connection(s)
I20260810 03:55:09.540673 3061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:55:09.541806 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 3061
I20260810 03:55:09.541922 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:55:09.542177 28261 external_mini_cluster.cc:1428] Setting key f968a85e8c35d67df024085a081e6f8f
I20260810 03:55:09.543843 3178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:55:09.550832 3178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ba07a0129e646aaae5cfcec225d5d19: Bootstrap starting.
Aug 10 03:55:09 dist-test-slave-pww0 krb5kdc[3041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:55:09.553176 3178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ba07a0129e646aaae5cfcec225d5d19: Neither blocks nor log segments found. Creating new log.
I20260810 03:55:09.553929 3178 log.cc:824] T 00000000000000000000000000000000 P 5ba07a0129e646aaae5cfcec225d5d19: Log is configured to *not* fsync() on all Append() calls
I20260810 03:55:09.556200 3178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ba07a0129e646aaae5cfcec225d5d19: No bootstrap required, opened a new log
I20260810 03:55:09.556433 3183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:09.543522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33735 (local address 127.27.153.126:40603)
0810 03:55:09.543932 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:09.543952 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:09.543998 (+ 46us) server_negotiation.cc:407] Connection header received
0810 03:55:09.544641 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:09.544657 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:09.545006 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:09.545288 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:09.546210 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:09.547193 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:09.548365 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:09.548832 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:09.548948 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:09.551693 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:09.551698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:09.551712 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:09.551741 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:09.553477 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:09.553911 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:09.553916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:09.553922 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:09.554036 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:09.554357 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:09.554360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:09.554362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:09.554890 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:09.555035 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:09.555331 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 03:55:09.555671 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":113,"threads_started":1}
I20260810 03:55:09.559319 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:55:09.560508 3178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ba07a0129e646aaae5cfcec225d5d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba07a0129e646aaae5cfcec225d5d19" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 40603 } }
I20260810 03:55:09.560825 3178 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ba07a0129e646aaae5cfcec225d5d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:55:09.560894 3178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ba07a0129e646aaae5cfcec225d5d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba07a0129e646aaae5cfcec225d5d19, State: Initialized, Role: FOLLOWER
I20260810 03:55:09.561376 3178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ba07a0129e646aaae5cfcec225d5d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba07a0129e646aaae5cfcec225d5d19" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 40603 } }
I20260810 03:55:09.561525 3178 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ba07a0129e646aaae5cfcec225d5d19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:55:09.561594 3178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ba07a0129e646aaae5cfcec225d5d19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:55:09.561698 3178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ba07a0129e646aaae5cfcec225d5d19 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:55:09.562857 3178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ba07a0129e646aaae5cfcec225d5d19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba07a0129e646aaae5cfcec225d5d19" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 40603 } }
I20260810 03:55:09.563234 3178 leader_election.cc:302] T 00000000000000000000000000000000 P 5ba07a0129e646aaae5cfcec225d5d19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ba07a0129e646aaae5cfcec225d5d19; no voters:
I20260810 03:55:09.563632 3178 leader_election.cc:288] T 00000000000000000000000000000000 P 5ba07a0129e646aaae5cfcec225d5d19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:55:09.563731 3195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ba07a0129e646aaae5cfcec225d5d19 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:55:09.563952 3195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ba07a0129e646aaae5cfcec225d5d19 [term 1 LEADER]: Becoming Leader. State: Replica: 5ba07a0129e646aaae5cfcec225d5d19, State: Running, Role: LEADER
I20260810 03:55:09.564365 3195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ba07a0129e646aaae5cfcec225d5d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba07a0129e646aaae5cfcec225d5d19" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 40603 } }
I20260810 03:55:09.564960 3178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ba07a0129e646aaae5cfcec225d5d19 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:55:09.565541 3198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba07a0129e646aaae5cfcec225d5d19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ba07a0129e646aaae5cfcec225d5d19. Latest consensus state: current_term: 1 leader_uuid: "5ba07a0129e646aaae5cfcec225d5d19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba07a0129e646aaae5cfcec225d5d19" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 40603 } } }
I20260810 03:55:09.565663 3198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba07a0129e646aaae5cfcec225d5d19 [sys.catalog]: This master's current role is: LEADER
I20260810 03:55:09.566048 3202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:55:09.566334 3197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba07a0129e646aaae5cfcec225d5d19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ba07a0129e646aaae5cfcec225d5d19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba07a0129e646aaae5cfcec225d5d19" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 40603 } } }
I20260810 03:55:09.566426 3197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba07a0129e646aaae5cfcec225d5d19 [sys.catalog]: This master's current role is: LEADER
I20260810 03:55:09.569453 3202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:55:09.572023 3183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:09.561657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39579 (local address 127.27.153.126:40603)
0810 03:55:09.561779 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:09.561783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:09.561941 (+ 158us) server_negotiation.cc:407] Connection header received
0810 03:55:09.562029 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:09.562032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:09.562080 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:09.562164 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:09.562764 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:09.563569 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:09.564652 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:09.565002 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:09.565101 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:09.568238 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:09.568242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:09.568245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:09.568272 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:09.569903 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:09.570404 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:09.570408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:09.570410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:09.570497 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:09.570784 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:09.570788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:09.570790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:09.571006 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:09.571134 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:09.571813 (+ 679us) server_negotiation.cc:299] Negotiation successful
0810 03:55:09.571915 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 03:55:09.575527 3202 catalog_manager.cc:1347] Generated new cluster ID: 35400208295d4b17a8082840f90d6854
I20260810 03:55:09.575588 3202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:55:09.592868 3202 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:55:09.593972 3202 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:55:09.611609 3202 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ba07a0129e646aaae5cfcec225d5d19: Generated new TSK 0
I20260810 03:55:09.612424 3202 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:55:09.625314 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 3061
2026-08-10T03:55:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1613 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:55:09 dist-test-slave-pww0 krb5kdc[3227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:55:09 dist-test-slave-pww0 krb5kdc[3227](info): set up 2 sockets
Aug 10 03:55:09 dist-test-slave-pww0 krb5kdc[3227](info): commencing operation
krb5kdc: starting...
W20260810 03:55:10.823973 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.169s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:55:10 dist-test-slave-pww0 krb5kdc[3227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:55:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:55:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:55:10.955705 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:39519
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:33071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:39519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:55:11.072754 3246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:55:11.073064 3246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:55:11.073127 3246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:55:11.077253 3246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:55:11.077324 3246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:55:11.077353 3246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:55:11.077373 3246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:55:11.077394 3246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:55:11.082906 3246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:33071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:39519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:39519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:55:11.084105 3246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:55:11.085201 3246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:55:11.094067 3256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:55:11.094238 3246 server_base.cc:1034] running on GCE node
W20260810 03:55:11.094077 3258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:11.094141 3261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:55:11.094996 3246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:55:11.096237 3246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:55:11.097398 3246 hybrid_clock.cc:648] HybridClock initialized: now 1786334111097379 us; error 43 us; skew 500 ppm
Aug 10 03:55:11 dist-test-slave-pww0 krb5kdc[3227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334111, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:55:11.100235 3246 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:55:11.101181 3246 webserver.cc:460] Webserver started at http://127.27.153.126:36403/ using document root <none> and password file <none>
I20260810 03:55:11.101738 3246 fs_manager.cc:359] Metadata directory not provided
I20260810 03:55:11.101826 3246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:55:11.102049 3246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:55:11.104305 3246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "3669babb8d294aec8d8cc08b3ba02c31"
format_stamp: "Formatted at 2026-08-10 03:55:11 on dist-test-slave-pww0"
server_key: "9f488fdb005097502fc5fe60aa269fe4"
server_key_iv: "bd6ae6e993ce1ddeebbdb84df52ae2a8"
server_key_version: "encryptionkey@0"
I20260810 03:55:11.104962 3246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "3669babb8d294aec8d8cc08b3ba02c31"
format_stamp: "Formatted at 2026-08-10 03:55:11 on dist-test-slave-pww0"
server_key: "9f488fdb005097502fc5fe60aa269fe4"
server_key_iv: "bd6ae6e993ce1ddeebbdb84df52ae2a8"
server_key_version: "encryptionkey@0"
I20260810 03:55:11.109051 3246 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:55:11.111850 3273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:55:11.113260 3246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 03:55:11.113411 3246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "3669babb8d294aec8d8cc08b3ba02c31"
format_stamp: "Formatted at 2026-08-10 03:55:11 on dist-test-slave-pww0"
server_key: "9f488fdb005097502fc5fe60aa269fe4"
server_key_iv: "bd6ae6e993ce1ddeebbdb84df52ae2a8"
server_key_version: "encryptionkey@0"
I20260810 03:55:11.113528 3246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:55:11.139117 3246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:55:11.142457 3246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:55:11.142659 3246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:55:11.149904 3246 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:39519
I20260810 03:55:11.149916 3365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:39519 every 8 connection(s)
I20260810 03:55:11.150931 3246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:55:11.153906 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 3246
I20260810 03:55:11.153930 3366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:55:11.154047 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:55:11.154341 28261 external_mini_cluster.cc:1428] Setting key b562a5f12a7abd7a05efd44a800cb5ce
I20260810 03:55:11.160995 3366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3669babb8d294aec8d8cc08b3ba02c31: Bootstrap starting.
I20260810 03:55:11.163836 3366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3669babb8d294aec8d8cc08b3ba02c31: Neither blocks nor log segments found. Creating new log.
I20260810 03:55:11.164263 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:11.155726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59077 (local address 127.27.153.126:39519)
0810 03:55:11.156182 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:11.156199 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:11.156236 (+ 37us) server_negotiation.cc:407] Connection header received
0810 03:55:11.157124 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:11.157144 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:11.157530 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:11.157823 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:11.158230 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:11.159251 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:11.160526 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:11.161007 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:11.161154 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:11.162248 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:11.162256 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:11.162275 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:55:11.162829 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:11.162893 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:55:11.162919 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:11.162999 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:11.163149 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 03:55:11.163365 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":133,"threads_started":1}
I20260810 03:55:11.164666 3366 log.cc:824] T 00000000000000000000000000000000 P 3669babb8d294aec8d8cc08b3ba02c31: Log is configured to *not* fsync() on all Append() calls
I20260810 03:55:11.166863 3366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3669babb8d294aec8d8cc08b3ba02c31: No bootstrap required, opened a new log
I20260810 03:55:11.167356 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:55:11.170948 3366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3669babb8d294aec8d8cc08b3ba02c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3669babb8d294aec8d8cc08b3ba02c31" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39519 } }
I20260810 03:55:11.171254 3366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3669babb8d294aec8d8cc08b3ba02c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:55:11.171329 3366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3669babb8d294aec8d8cc08b3ba02c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3669babb8d294aec8d8cc08b3ba02c31, State: Initialized, Role: FOLLOWER
I20260810 03:55:11.171868 3366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3669babb8d294aec8d8cc08b3ba02c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3669babb8d294aec8d8cc08b3ba02c31" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39519 } }
I20260810 03:55:11.172039 3366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3669babb8d294aec8d8cc08b3ba02c31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:55:11.172116 3366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3669babb8d294aec8d8cc08b3ba02c31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:55:11.172231 3366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3669babb8d294aec8d8cc08b3ba02c31 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:55:11.173488 3366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3669babb8d294aec8d8cc08b3ba02c31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3669babb8d294aec8d8cc08b3ba02c31" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39519 } }
I20260810 03:55:11.173911 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:11.169417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35911 (local address 127.27.153.126:39519)
0810 03:55:11.169534 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:11.169539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:11.169722 (+ 183us) server_negotiation.cc:407] Connection header received
0810 03:55:11.169853 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:11.169855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:11.169895 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:11.169962 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:11.170268 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:11.170974 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:11.172171 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:11.172472 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:11.172564 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:11.173433 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:11.173436 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:11.173438 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:55:11.173450 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:11.173478 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:55:11.173488 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:11.173555 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:11.173711 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 03:55:11.173771 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 03:55:11.173894 3366 leader_election.cc:302] T 00000000000000000000000000000000 P 3669babb8d294aec8d8cc08b3ba02c31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3669babb8d294aec8d8cc08b3ba02c31; no voters:
I20260810 03:55:11.174311 3366 leader_election.cc:288] T 00000000000000000000000000000000 P 3669babb8d294aec8d8cc08b3ba02c31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:55:11.174391 3392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3669babb8d294aec8d8cc08b3ba02c31 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:55:11.174602 3392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3669babb8d294aec8d8cc08b3ba02c31 [term 1 LEADER]: Becoming Leader. State: Replica: 3669babb8d294aec8d8cc08b3ba02c31, State: Running, Role: LEADER
I20260810 03:55:11.174990 3392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3669babb8d294aec8d8cc08b3ba02c31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3669babb8d294aec8d8cc08b3ba02c31" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39519 } }
I20260810 03:55:11.175782 3366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3669babb8d294aec8d8cc08b3ba02c31 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:55:11.176719 3393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3669babb8d294aec8d8cc08b3ba02c31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3669babb8d294aec8d8cc08b3ba02c31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3669babb8d294aec8d8cc08b3ba02c31" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39519 } } }
I20260810 03:55:11.176865 3393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3669babb8d294aec8d8cc08b3ba02c31 [sys.catalog]: This master's current role is: LEADER
I20260810 03:55:11.177170 3395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3669babb8d294aec8d8cc08b3ba02c31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3669babb8d294aec8d8cc08b3ba02c31. Latest consensus state: current_term: 1 leader_uuid: "3669babb8d294aec8d8cc08b3ba02c31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3669babb8d294aec8d8cc08b3ba02c31" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 39519 } } }
I20260810 03:55:11.177248 3395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3669babb8d294aec8d8cc08b3ba02c31 [sys.catalog]: This master's current role is: LEADER
I20260810 03:55:11.177284 3404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:55:11.180450 3404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:55:11.186522 3404 catalog_manager.cc:1347] Generated new cluster ID: 186aaecb5e544ef7aac18d913bf291e6
I20260810 03:55:11.186611 3404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:55:11.196029 3404 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:55:11.197073 3404 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:55:11.210440 3404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3669babb8d294aec8d8cc08b3ba02c31: Generated new TSK 0
I20260810 03:55:11.211375 3404 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:55:11.218490 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 3246
2026-08-10T03:55:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1599 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:55:11 dist-test-slave-pww0 krb5kdc[3423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:55:11 dist-test-slave-pww0 krb5kdc[3423](info): set up 2 sockets
Aug 10 03:55:11 dist-test-slave-pww0 krb5kdc[3423](info): commencing operation
krb5kdc: starting...
W20260810 03:55:12.420645 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:55:12 dist-test-slave-pww0 krb5kdc[3423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:55:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:55:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:55:12.552882 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:44833
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:36913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:44833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:55:12.668632 3445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:55:12.668936 3445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:55:12.669000 3445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:55:12.673166 3445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:55:12.673243 3445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:55:12.673278 3445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:55:12.673301 3445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:55:12.673331 3445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:55:12.679014 3445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:36913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:44833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:44833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:55:12.680230 3445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:55:12.681340 3445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:55:12.690434 3451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:55:12.690481 3445 server_base.cc:1034] running on GCE node
W20260810 03:55:12.690426 3452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:12.690426 3456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:55:12.691401 3445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:55:12.692636 3445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:55:12.693817 3445 hybrid_clock.cc:648] HybridClock initialized: now 1786334112693793 us; error 38 us; skew 500 ppm
Aug 10 03:55:12 dist-test-slave-pww0 krb5kdc[3423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334112, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:55:12.701171 3445 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:55:12.702237 3445 webserver.cc:460] Webserver started at http://127.27.153.126:33803/ using document root <none> and password file <none>
I20260810 03:55:12.702839 3445 fs_manager.cc:359] Metadata directory not provided
I20260810 03:55:12.702916 3445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:55:12.703145 3445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:55:12.705442 3445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "c2cbaf1aa13d4ddeb84ca25e66bf9124"
format_stamp: "Formatted at 2026-08-10 03:55:12 on dist-test-slave-pww0"
server_key: "638a97ebf370497afbeb1e1cab52316a"
server_key_iv: "3b1f35c68297077b6d922627f3890bcf"
server_key_version: "encryptionkey@0"
I20260810 03:55:12.706051 3445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "c2cbaf1aa13d4ddeb84ca25e66bf9124"
format_stamp: "Formatted at 2026-08-10 03:55:12 on dist-test-slave-pww0"
server_key: "638a97ebf370497afbeb1e1cab52316a"
server_key_iv: "3b1f35c68297077b6d922627f3890bcf"
server_key_version: "encryptionkey@0"
I20260810 03:55:12.710264 3445 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 03:55:12.713011 3466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:55:12.714397 3445 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 03:55:12.714530 3445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "c2cbaf1aa13d4ddeb84ca25e66bf9124"
format_stamp: "Formatted at 2026-08-10 03:55:12 on dist-test-slave-pww0"
server_key: "638a97ebf370497afbeb1e1cab52316a"
server_key_iv: "3b1f35c68297077b6d922627f3890bcf"
server_key_version: "encryptionkey@0"
I20260810 03:55:12.714641 3445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:55:12.719704 3445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:55:12.723167 3445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:55:12.723352 3445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:55:12.730388 3445 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:44833
I20260810 03:55:12.730399 3552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:44833 every 8 connection(s)
I20260810 03:55:12.731383 3445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:55:12.734256 3554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:55:12.740157 3554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2cbaf1aa13d4ddeb84ca25e66bf9124: Bootstrap starting.
I20260810 03:55:12.741145 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 3445
I20260810 03:55:12.741288 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:55:12.741537 28261 external_mini_cluster.cc:1428] Setting key 49a0bdc1d95a6350d1c1343681781b40
I20260810 03:55:12.742990 3554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2cbaf1aa13d4ddeb84ca25e66bf9124: Neither blocks nor log segments found. Creating new log.
I20260810 03:55:12.743813 3554 log.cc:824] T 00000000000000000000000000000000 P c2cbaf1aa13d4ddeb84ca25e66bf9124: Log is configured to *not* fsync() on all Append() calls
I20260810 03:55:12.745940 3554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2cbaf1aa13d4ddeb84ca25e66bf9124: No bootstrap required, opened a new log
I20260810 03:55:12.746750 3562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:12.742681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37661 (local address 127.27.153.126:44833)
0810 03:55:12.743220 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:12.743236 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:12.743272 (+ 36us) server_negotiation.cc:407] Connection header received
0810 03:55:12.743979 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:12.743999 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:12.744358 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:12.744738 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:12.745508 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:12.745513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:12.745525 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:55:12.745535 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:12.745588 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:55:12.745607 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:12.745706 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:12.745837 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 03:55:12.746001 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":156,"threads_started":1}
I20260810 03:55:12.749550 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:55:12.749816 3554 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2cbaf1aa13d4ddeb84ca25e66bf9124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2cbaf1aa13d4ddeb84ca25e66bf9124" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 44833 } }
I20260810 03:55:12.750046 3554 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2cbaf1aa13d4ddeb84ca25e66bf9124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 03:55:12.750098 3554 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2cbaf1aa13d4ddeb84ca25e66bf9124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2cbaf1aa13d4ddeb84ca25e66bf9124, State: Initialized, Role: FOLLOWER
I20260810 03:55:12.750538 3554 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2cbaf1aa13d4ddeb84ca25e66bf9124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2cbaf1aa13d4ddeb84ca25e66bf9124" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 44833 } }
I20260810 03:55:12.750665 3554 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2cbaf1aa13d4ddeb84ca25e66bf9124 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 03:55:12.750726 3554 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2cbaf1aa13d4ddeb84ca25e66bf9124 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 03:55:12.750828 3554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2cbaf1aa13d4ddeb84ca25e66bf9124 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:55:12.751930 3554 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2cbaf1aa13d4ddeb84ca25e66bf9124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2cbaf1aa13d4ddeb84ca25e66bf9124" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 44833 } }
I20260810 03:55:12.752274 3554 leader_election.cc:302] T 00000000000000000000000000000000 P c2cbaf1aa13d4ddeb84ca25e66bf9124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2cbaf1aa13d4ddeb84ca25e66bf9124; no voters:
I20260810 03:55:12.752609 3554 leader_election.cc:288] T 00000000000000000000000000000000 P c2cbaf1aa13d4ddeb84ca25e66bf9124 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 03:55:12.752674 3578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2cbaf1aa13d4ddeb84ca25e66bf9124 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:55:12.752944 3578 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2cbaf1aa13d4ddeb84ca25e66bf9124 [term 1 LEADER]: Becoming Leader. State: Replica: c2cbaf1aa13d4ddeb84ca25e66bf9124, State: Running, Role: LEADER
I20260810 03:55:12.753343 3578 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2cbaf1aa13d4ddeb84ca25e66bf9124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2cbaf1aa13d4ddeb84ca25e66bf9124" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 44833 } }
I20260810 03:55:12.753479 3562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:12.751706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35735 (local address 127.27.153.126:44833)
0810 03:55:12.751835 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:12.751840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:12.752060 (+ 220us) server_negotiation.cc:407] Connection header received
0810 03:55:12.752178 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:12.752181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:12.752219 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:12.752285 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:12.753059 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:12.753062 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:12.753065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 03:55:12.753078 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:12.753108 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 03:55:12.753120 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:12.753203 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:12.753323 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 03:55:12.753363 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 03:55:12.754043 3554 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2cbaf1aa13d4ddeb84ca25e66bf9124 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:55:12.755054 3579 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2cbaf1aa13d4ddeb84ca25e66bf9124 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2cbaf1aa13d4ddeb84ca25e66bf9124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2cbaf1aa13d4ddeb84ca25e66bf9124" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 44833 } } }
I20260810 03:55:12.755198 3579 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2cbaf1aa13d4ddeb84ca25e66bf9124 [sys.catalog]: This master's current role is: LEADER
I20260810 03:55:12.755435 3579 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2cbaf1aa13d4ddeb84ca25e66bf9124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2cbaf1aa13d4ddeb84ca25e66bf9124. Latest consensus state: current_term: 1 leader_uuid: "c2cbaf1aa13d4ddeb84ca25e66bf9124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2cbaf1aa13d4ddeb84ca25e66bf9124" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 44833 } } }
I20260810 03:55:12.755600 3595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:55:12.755601 3579 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2cbaf1aa13d4ddeb84ca25e66bf9124 [sys.catalog]: This master's current role is: LEADER
I20260810 03:55:12.758569 3595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:55:12.764436 3595 catalog_manager.cc:1347] Generated new cluster ID: 897ebcae53ee4daa9018c0fa39d2acf5
I20260810 03:55:12.764524 3595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:55:12.785058 3595 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:55:12.785441 3595 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:55:12.793684 3595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2cbaf1aa13d4ddeb84ca25e66bf9124: Generated new TSK 0
I20260810 03:55:12.794433 3595 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 03:55:12.804337 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 3445
2026-08-10T03:55:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1590 ms)
[----------] 18 tests from AuthTokenIssuingTest (28753 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:55:12 dist-test-slave-pww0 krb5kdc[3618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:55:12 dist-test-slave-pww0 krb5kdc[3618](info): set up 2 sockets
Aug 10 03:55:12 dist-test-slave-pww0 krb5kdc[3618](info): commencing operation
krb5kdc: starting...
W20260810 03:55:14.018747 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:55:14 dist-test-slave-pww0 krb5kdc[3618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:55:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:55:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:55:14.152410 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:33399
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:33399,127.27.153.125:43115,127.27.153.124:34483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:55:14.268477 3639 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:55:14.268805 3639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:55:14.268836 3639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:55:14.268888 3639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:55:14.273099 3639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:55:14.273183 3639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:55:14.273201 3639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:55:14.273217 3639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:55:14.273231 3639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:55:14.278792 3639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:33399,127.27.153.125:43115,127.27.153.124:34483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:33399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:55:14.280045 3639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:55:14.281088 3639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:55:14.290032 3647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:14.290032 3651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:14.290222 3649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:55:14.290350 3639 server_base.cc:1034] running on GCE node
I20260810 03:55:14.291054 3639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:55:14.292320 3639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:55:14.293490 3639 hybrid_clock.cc:648] HybridClock initialized: now 1786334114293470 us; error 34 us; skew 500 ppm
Aug 10 03:55:14 dist-test-slave-pww0 krb5kdc[3618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334114, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:55:14.296414 3639 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:55:14.297428 3639 webserver.cc:460] Webserver started at http://127.27.153.126:39383/ using document root <none> and password file <none>
I20260810 03:55:14.298010 3639 fs_manager.cc:359] Metadata directory not provided
I20260810 03:55:14.298084 3639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:55:14.298313 3639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:55:14.300611 3639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "464fb81536324a02ad6318f54a7b7ee1"
format_stamp: "Formatted at 2026-08-10 03:55:14 on dist-test-slave-pww0"
server_key: "76f55be292813db9c65d5779e8997da9"
server_key_iv: "40f288fd357442e0d1cd0e37b3927953"
server_key_version: "encryptionkey@0"
I20260810 03:55:14.301214 3639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "464fb81536324a02ad6318f54a7b7ee1"
format_stamp: "Formatted at 2026-08-10 03:55:14 on dist-test-slave-pww0"
server_key: "76f55be292813db9c65d5779e8997da9"
server_key_iv: "40f288fd357442e0d1cd0e37b3927953"
server_key_version: "encryptionkey@0"
I20260810 03:55:14.305318 3639 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 03:55:14.308007 3660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:55:14.309289 3639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:55:14.309423 3639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "464fb81536324a02ad6318f54a7b7ee1"
format_stamp: "Formatted at 2026-08-10 03:55:14 on dist-test-slave-pww0"
server_key: "76f55be292813db9c65d5779e8997da9"
server_key_iv: "40f288fd357442e0d1cd0e37b3927953"
server_key_version: "encryptionkey@0"
I20260810 03:55:14.309536 3639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:55:14.327976 3639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:55:14.331881 3639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:55:14.332098 3639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:55:14.339491 3639 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:33399
I20260810 03:55:14.339891 3748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:33399 every 8 connection(s)
I20260810 03:55:14.340559 3639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:55:14.343750 3749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:55:14.348181 3749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33399 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:55:14.350590 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 3639
I20260810 03:55:14.350706 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:55:14.350991 28261 external_mini_cluster.cc:1428] Setting key 5cdf71c8b8ab1793ec777d53c2b35783
Aug 10 03:55:14 dist-test-slave-pww0 krb5kdc[3618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334114, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.153.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.125@KRBTEST.COM" created.
I20260810 03:55:14.364795 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.350744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:33399 (local address 127.0.0.1:39900)
0810 03:55:14.351416 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.351520 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0810 03:55:14.352637 (+ 1117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:55:14.353449 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:55:14.353455 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:55:14.353957 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:55:14.354226 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:14.354241 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.355436 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.355439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:14.356336 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:14.356348 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.357013 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.357017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:14.357133 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.357787 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:55:14.357806 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:55:14.359287 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:55:14.361327 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:14.361337 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:14.361349 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:14.361616 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:14.361979 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:14.361982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:14.361984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:14.362107 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:14.362783 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:55:14.362790 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:55:14.362944 (+ 154us) client_negotiation.cc:769] Sending connection context
0810 03:55:14.363105 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 03:55:14.363583 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":158,"thread_start_us":156,"threads_started":1}
I20260810 03:55:14.364830 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.351781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39900 (local address 127.27.153.126:33399)
0810 03:55:14.352093 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:14.352104 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:14.352149 (+ 45us) server_negotiation.cc:407] Connection header received
0810 03:55:14.353008 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:14.353013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:14.353231 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:14.353417 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:14.354382 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.355304 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.356457 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.356851 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.357012 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.359434 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:14.359440 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:14.359452 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:14.359503 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:14.361213 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.361724 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.361729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.361730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.361881 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.362189 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.362192 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.362194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.362677 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:14.362807 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:14.363196 (+ 389us) server_negotiation.cc:299] Negotiation successful
0810 03:55:14.363528 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":199,"thread_start_us":94,"threads_started":1}
I20260810 03:55:14.367825 3749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:55:14.368813 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.368476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:43115 (local address 127.0.0.1:44336)
0810 03:55:14.368639 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.368731 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.125:43115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260810 03:55:14.369131 3667 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.153.125:43115: connect: Connection refused (error 111)
I20260810 03:55:14.370263 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.369959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:43115 (local address 127.0.0.1:44350)
0810 03:55:14.370124 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.370184 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.125:43115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260810 03:55:14.370440 3749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.125:43115: Network error: Client connection negotiation failed: client connection to 127.27.153.125:43115: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.125.keytab.
Entry for principal kudu/127.27.153.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.125.keytab.
I20260810 03:55:14.380450 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.125:43115
--webserver_interface=127.27.153.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.125.keytab
--principal=kudu/127.27.153.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:33399,127.27.153.125:43115,127.27.153.124:34483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:55:14.419725 3749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } attempt: 1
I20260810 03:55:14.420593 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.420291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:43115 (local address 127.0.0.1:44360)
0810 03:55:14.420446 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.420514 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.125:43115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 03:55:14.421423 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.421180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:43115 (local address 127.0.0.1:44364)
0810 03:55:14.421301 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.421350 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.125:43115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 03:55:14.421571 3749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.125:43115: Network error: Client connection negotiation failed: client connection to 127.27.153.125:43115: connect: Connection refused (error 111)
I20260810 03:55:14.489759 3749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } attempt: 2
I20260810 03:55:14.490788 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.490430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:43115 (local address 127.0.0.1:44366)
0810 03:55:14.490626 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.490699 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.125:43115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 03:55:14.491621 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.491349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:43115 (local address 127.0.0.1:44370)
0810 03:55:14.491450 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.491549 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.125:43115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 03:55:14.491757 3749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.125:43115: Network error: Client connection negotiation failed: client connection to 127.27.153.125:43115: connect: Connection refused (error 111)
W20260810 03:55:14.497973 3767 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:55:14.498337 3767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:55:14.498382 3767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:55:14.498451 3767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:55:14.502853 3767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:55:14.502941 3767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:55:14.502969 3767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:55:14.502990 3767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:55:14.503012 3767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:55:14.509502 3767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:33399,127.27.153.125:43115,127.27.153.124:34483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.125.keytab
--principal=kudu/127.27.153.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.125:43115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.153.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:55:14.510751 3767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:55:14.511922 3767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:55:14.522305 3773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:14.522305 3774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:14.522305 3778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:55:14.528410 3767 server_base.cc:1034] running on GCE node
I20260810 03:55:14.529354 3767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:55:14.530942 3767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:55:14.532120 3767 hybrid_clock.cc:648] HybridClock initialized: now 1786334114532099 us; error 45 us; skew 500 ppm
Aug 10 03:55:14 dist-test-slave-pww0 krb5kdc[3618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334114, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:55:14.535166 3767 init.cc:377] Logged in from keytab as kudu/127.27.153.125@KRBTEST.COM (short username kudu)
I20260810 03:55:14.536267 3767 webserver.cc:460] Webserver started at http://127.27.153.125:32923/ using document root <none> and password file <none>
I20260810 03:55:14.536854 3767 fs_manager.cc:359] Metadata directory not provided
I20260810 03:55:14.536923 3767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:55:14.537135 3767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:55:14.539423 3767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-1/data/instance:
uuid: "d7627b3decad46bb8be3dcf42dc951c5"
format_stamp: "Formatted at 2026-08-10 03:55:14 on dist-test-slave-pww0"
server_key: "56d85749d716764d90bf60b030275b57"
server_key_iv: "3285ad415400f918993f25bfcfb04b4a"
server_key_version: "encryptionkey@0"
I20260810 03:55:14.540078 3767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-1/wal/instance:
uuid: "d7627b3decad46bb8be3dcf42dc951c5"
format_stamp: "Formatted at 2026-08-10 03:55:14 on dist-test-slave-pww0"
server_key: "56d85749d716764d90bf60b030275b57"
server_key_iv: "3285ad415400f918993f25bfcfb04b4a"
server_key_version: "encryptionkey@0"
I20260810 03:55:14.544196 3767 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 03:55:14.546864 3789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:55:14.548348 3767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:55:14.548506 3767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-1/wal
uuid: "d7627b3decad46bb8be3dcf42dc951c5"
format_stamp: "Formatted at 2026-08-10 03:55:14 on dist-test-slave-pww0"
server_key: "56d85749d716764d90bf60b030275b57"
server_key_iv: "3285ad415400f918993f25bfcfb04b4a"
server_key_version: "encryptionkey@0"
I20260810 03:55:14.548632 3767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:55:14.582931 3749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } attempt: 3
I20260810 03:55:14.583906 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.583598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:43115 (local address 127.0.0.1:44380)
0810 03:55:14.583752 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.583824 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.125:43115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 03:55:14.584620 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.584402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:43115 (local address 127.0.0.1:44382)
0810 03:55:14.584505 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.584551 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.125:43115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 03:55:14.584739 3749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.125:43115: Network error: Client connection negotiation failed: client connection to 127.27.153.125:43115: connect: Connection refused (error 111)
I20260810 03:55:14.595768 3767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:55:14.599089 3767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:55:14.599308 3767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:55:14.606551 3767 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.125:43115
I20260810 03:55:14.606858 3869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.125:43115 every 8 connection(s)
I20260810 03:55:14.607714 3767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-1/data/info.pb
I20260810 03:55:14.610718 3870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:55:14.610939 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 3767
I20260810 03:55:14.611069 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-1/wal/instance
I20260810 03:55:14.611344 28261 external_mini_cluster.cc:1428] Setting key 7cf27d63fd3c5c67ba954a9a1a0d717d
I20260810 03:55:14.615105 3870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33399 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.153.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.124@KRBTEST.COM" created.
Aug 10 03:55:14 dist-test-slave-pww0 krb5kdc[3618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334114, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.125@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:55:14.629359 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.617217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39912 (local address 127.27.153.126:33399)
0810 03:55:14.617352 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:14.617355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:14.618548 (+ 1193us) server_negotiation.cc:407] Connection header received
0810 03:55:14.619535 (+ 987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:14.619538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:14.619599 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:14.619683 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:14.620911 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.621465 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.622624 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.622873 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.622974 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.625575 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:14.625579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:14.625581 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:14.625611 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:14.626993 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.627689 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.627692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.627693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.627803 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.628158 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.628162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.628163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.628369 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:14.628462 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:14.629034 (+ 572us) server_negotiation.cc:299] Negotiation successful
0810 03:55:14.629186 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 03:55:14.630247 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.617843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:33399 (local address 127.0.0.1:39912)
0810 03:55:14.618388 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.618434 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 03:55:14.619301 (+ 867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:55:14.619868 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:55:14.619875 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:55:14.620257 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:55:14.620759 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:14.620774 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.621594 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.621597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:14.622484 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:14.622496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.623037 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.623041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:14.623219 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.623814 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:55:14.623832 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:55:14.625370 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:55:14.627139 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:14.627146 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:14.627165 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:14.627577 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:14.627900 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:14.627904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:14.627906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:14.628067 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:14.628476 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:55:14.628484 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:55:14.628775 (+ 291us) client_negotiation.cc:769] Sending connection context
0810 03:55:14.628961 (+ 186us) client_negotiation.cc:240] Negotiation successful
0810 03:55:14.629337 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":142,"threads_started":1}
I20260810 03:55:14.631937 3870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:55:14 dist-test-slave-pww0 krb5kdc[3618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334114, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.125@KRBTEST.COM for kudu/127.27.153.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.124.keytab.
Entry for principal kudu/127.27.153.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.124.keytab.
I20260810 03:55:14.641036 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.124:34483
--webserver_interface=127.27.153.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.124.keytab
--principal=kudu/127.27.153.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:33399,127.27.153.125:43115,127.27.153.124:34483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:55:14.642264 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.632505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:43115 (local address 127.0.0.1:44386)
0810 03:55:14.632705 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.632721 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 03:55:14.632823 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:55:14.633516 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:55:14.633519 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:55:14.633744 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:55:14.633858 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:14.633864 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.634940 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.634943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:14.635568 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:14.635578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.636234 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.636239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:14.636358 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.637026 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:55:14.637038 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:55:14.638389 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:55:14.640503 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:14.640506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:14.640507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:14.640733 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:14.641054 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:14.641056 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:14.641058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:14.641132 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:14.641814 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:55:14.641817 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:55:14.641914 (+ 97us) client_negotiation.cc:769] Sending connection context
0810 03:55:14.642004 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 03:55:14.642126 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":26}
I20260810 03:55:14.642467 3883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.632658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44386 (local address 127.27.153.125:43115)
0810 03:55:14.633054 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:14.633064 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:14.633085 (+ 21us) server_negotiation.cc:407] Connection header received
0810 03:55:14.633140 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:14.633145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:14.633367 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:14.633490 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:14.633961 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.634808 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.635692 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.636105 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.636223 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.638538 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:14.638561 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:14.638577 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:14.638601 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:14.640368 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.640855 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.640862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.640864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.640969 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.641319 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.641322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.641324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.641692 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:14.641843 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:14.642112 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 03:55:14.642293 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":299,"thread_start_us":186,"threads_started":1}
I20260810 03:55:14.644735 3870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:55:14.645612 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.645261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:34483 (local address 127.0.0.1:41042)
0810 03:55:14.645449 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.645538 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:34483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260810 03:55:14.645861 3790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.153.124:34483: connect: Connection refused (error 111)
I20260810 03:55:14.646854 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.646642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:34483 (local address 127.0.0.1:41058)
0810 03:55:14.646745 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.646795 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:34483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 03:55:14.647002 3870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.124:34483: Network error: Client connection negotiation failed: client connection to 127.27.153.124:34483: connect: Connection refused (error 111)
I20260810 03:55:14.696270 3870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } attempt: 1
I20260810 03:55:14.697181 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.696882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:34483 (local address 127.0.0.1:41066)
0810 03:55:14.697029 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.697098 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:34483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 03:55:14.697989 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.697741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:34483 (local address 127.0.0.1:41078)
0810 03:55:14.697848 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.697908 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:34483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 03:55:14.698153 3870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.124:34483: Network error: Client connection negotiation failed: client connection to 127.27.153.124:34483: connect: Connection refused (error 111)
I20260810 03:55:14.727895 3749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } attempt: 4
Aug 10 03:55:14 dist-test-slave-pww0 krb5kdc[3618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334114, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for kudu/127.27.153.125@KRBTEST.COM
I20260810 03:55:14.738200 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.728485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:43115 (local address 127.0.0.1:44394)
0810 03:55:14.728675 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.728692 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 03:55:14.728794 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:55:14.729078 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:55:14.729081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:55:14.729261 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:55:14.729381 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:14.729388 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.730888 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.730891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:14.731998 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:14.732011 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.732675 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.732679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:14.732771 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.733286 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:55:14.733297 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:55:14.734777 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:55:14.736663 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:14.736666 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:14.736668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:14.736863 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:14.737163 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:14.737165 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:14.737167 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:14.737246 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:14.737696 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:55:14.737698 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:55:14.737798 (+ 100us) client_negotiation.cc:769] Sending connection context
0810 03:55:14.737896 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 03:55:14.738066 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260810 03:55:14.738278 3883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.728582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44394 (local address 127.27.153.125:43115)
0810 03:55:14.728744 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:14.728749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:14.728768 (+ 19us) server_negotiation.cc:407] Connection header received
0810 03:55:14.728883 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:14.728887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:14.728944 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:14.729040 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:14.729510 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.730739 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.732129 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.732519 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.732646 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.734937 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:14.734942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:14.734945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:14.734977 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:14.736556 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.736954 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.736958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.736960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.737058 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.737324 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.737327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.737329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.737572 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:14.737679 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:14.737970 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 03:55:14.738109 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 03:55:14.739279 3749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:55:14.740056 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.739816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:34483 (local address 127.0.0.1:41088)
0810 03:55:14.739935 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.739988 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:34483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 03:55:14.740726 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.740536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:34483 (local address 127.0.0.1:41098)
0810 03:55:14.740631 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.740676 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:34483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 03:55:14.740875 3749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.124:34483: Network error: Client connection negotiation failed: client connection to 127.27.153.124:34483: connect: Connection refused (error 111)
W20260810 03:55:14.761386 3885 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:55:14.761686 3885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:55:14.761709 3885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:55:14.761754 3885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:55:14.765785 3885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:55:14.765854 3885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:55:14.765872 3885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:55:14.765887 3885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:55:14.765908 3885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:55:14.766300 3870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } attempt: 2
I20260810 03:55:14.767168 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.766857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:34483 (local address 127.0.0.1:41104)
0810 03:55:14.767021 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.767086 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:34483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 03:55:14.767864 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.767655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:34483 (local address 127.0.0.1:41110)
0810 03:55:14.767755 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.767804 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:34483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 03:55:14.767976 3870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.124:34483: Network error: Client connection negotiation failed: client connection to 127.27.153.124:34483: connect: Connection refused (error 111)
I20260810 03:55:14.771829 3885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:33399,127.27.153.125:43115,127.27.153.124:34483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.124.keytab
--principal=kudu/127.27.153.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.124:34483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.153.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:55:14.773003 3885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:55:14.774078 3885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:55:14.783174 3900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:14.783306 3897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:14.783180 3895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:55:14.784431 3885 server_base.cc:1034] running on GCE node
I20260810 03:55:14.785064 3885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:55:14.786221 3885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:55:14.787400 3885 hybrid_clock.cc:648] HybridClock initialized: now 1786334114787374 us; error 36 us; skew 500 ppm
Aug 10 03:55:14 dist-test-slave-pww0 krb5kdc[3618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334114, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:55:14.790221 3885 init.cc:377] Logged in from keytab as kudu/127.27.153.124@KRBTEST.COM (short username kudu)
I20260810 03:55:14.791200 3885 webserver.cc:460] Webserver started at http://127.27.153.124:45999/ using document root <none> and password file <none>
I20260810 03:55:14.791813 3885 fs_manager.cc:359] Metadata directory not provided
I20260810 03:55:14.791880 3885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:55:14.792059 3885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:55:14.794215 3885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-2/data/instance:
uuid: "152b4fbeee314501b596bc0273795427"
format_stamp: "Formatted at 2026-08-10 03:55:14 on dist-test-slave-pww0"
server_key: "cfdaf020678462421817319e707ea921"
server_key_iv: "3521fb92327105f364b7a207dfaf98ea"
server_key_version: "encryptionkey@0"
I20260810 03:55:14.794793 3885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-2/wal/instance:
uuid: "152b4fbeee314501b596bc0273795427"
format_stamp: "Formatted at 2026-08-10 03:55:14 on dist-test-slave-pww0"
server_key: "cfdaf020678462421817319e707ea921"
server_key_iv: "3521fb92327105f364b7a207dfaf98ea"
server_key_version: "encryptionkey@0"
I20260810 03:55:14.798882 3885 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260810 03:55:14.800055 3749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } attempt: 1
I20260810 03:55:14.800884 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.800572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:34483 (local address 127.0.0.1:41124)
0810 03:55:14.800726 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.800803 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:34483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 03:55:14.801682 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.801430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:34483 (local address 127.0.0.1:41130)
0810 03:55:14.801546 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.801606 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:34483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 03:55:14.801718 3909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20260810 03:55:14.801826 3749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.124:34483: Network error: Client connection negotiation failed: client connection to 127.27.153.124:34483: connect: Connection refused (error 111)
I20260810 03:55:14.803057 3885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 03:55:14.803192 3885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-2/wal
uuid: "152b4fbeee314501b596bc0273795427"
format_stamp: "Formatted at 2026-08-10 03:55:14 on dist-test-slave-pww0"
server_key: "cfdaf020678462421817319e707ea921"
server_key_iv: "3521fb92327105f364b7a207dfaf98ea"
server_key_version: "encryptionkey@0"
I20260810 03:55:14.803308 3885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:55:14.823676 3885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:55:14.827291 3885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:55:14.827529 3885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:55:14.834736 3885 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.124:34483
I20260810 03:55:14.835014 4000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.124:34483 every 8 connection(s)
I20260810 03:55:14.835814 3885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-2/data/info.pb
I20260810 03:55:14.838761 4001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:55:14.839798 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 3885
I20260810 03:55:14.839931 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786333814837068-28261-0/minicluster-data/master-2/wal/instance
I20260810 03:55:14.840251 28261 external_mini_cluster.cc:1428] Setting key e5f0da0a4dae4868323d1bb45a54830b
I20260810 03:55:14.843520 4001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33399 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:55:14 dist-test-slave-pww0 krb5kdc[3618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.124@KRBTEST.COM
Aug 10 03:55:14 dist-test-slave-pww0 krb5kdc[3618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334114, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.124@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:55:14.854795 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.845425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39928 (local address 127.27.153.126:33399)
0810 03:55:14.845589 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:14.845595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:14.845979 (+ 384us) server_negotiation.cc:407] Connection header received
0810 03:55:14.846156 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:14.846159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:14.846207 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:14.846297 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:14.847012 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.847551 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.848407 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.848710 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.848813 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.851367 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:14.851370 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:14.851372 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:14.851401 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:14.852732 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.853219 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.853222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.853223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.853305 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.853704 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.853707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.853708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.853988 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:14.854097 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:14.854490 (+ 393us) server_negotiation.cc:299] Negotiation successful
0810 03:55:14.854624 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 03:55:14.855829 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.845362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:33399 (local address 127.0.0.1:39928)
0810 03:55:14.845832 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.845887 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 03:55:14.846020 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:55:14.846350 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:55:14.846355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:55:14.846762 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:55:14.846889 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:14.846895 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.847663 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.847666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:14.848291 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:14.848301 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.848857 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.848861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:14.848959 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.849429 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:55:14.849446 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:55:14.851213 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:55:14.852839 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:14.852850 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:14.852861 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:14.853091 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:14.853392 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:14.853395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:14.853397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:14.853607 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:14.854091 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:55:14.854097 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:55:14.854249 (+ 152us) client_negotiation.cc:769] Sending connection context
0810 03:55:14.854423 (+ 174us) client_negotiation.cc:240] Negotiation successful
0810 03:55:14.854785 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"spinlock_wait_cycles":5120,"thread_start_us":153,"threads_started":1}
I20260810 03:55:14.855831 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.841923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41136 (local address 127.27.153.124:34483)
0810 03:55:14.842474 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:14.842491 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:14.842535 (+ 44us) server_negotiation.cc:407] Connection header received
0810 03:55:14.843250 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:14.843270 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:14.843708 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:14.844106 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:14.845073 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.846434 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.847417 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.847980 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.848135 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.850888 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:14.850895 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:14.850910 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:14.850939 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:14.852797 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.853297 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.853303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.853304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.853418 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.853731 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.853734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.853735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.854243 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:14.854408 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:14.854698 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 03:55:14.854846 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":176,"threads_started":1}
I20260810 03:55:14.857100 4001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:55:14.859148 3870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } attempt: 3
Aug 10 03:55:14 dist-test-slave-pww0 krb5kdc[3618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334114, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.124@KRBTEST.COM for kudu/127.27.153.125@KRBTEST.COM
Aug 10 03:55:14 dist-test-slave-pww0 krb5kdc[3618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
Aug 10 03:55:14 dist-test-slave-pww0 krb5kdc[3618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334114, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.125@KRBTEST.COM for kudu/127.27.153.124@KRBTEST.COM
I20260810 03:55:14.867125 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.857632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:43115 (local address 127.0.0.1:44396)
0810 03:55:14.857800 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.857814 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:55:14.857884 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:55:14.858105 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:55:14.858108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:55:14.858386 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:55:14.858512 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:14.858519 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.859524 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.859527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:14.860255 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:14.860264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.860714 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.860718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:14.860817 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.862043 (+ 1226us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:55:14.862055 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:55:14.863814 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:55:14.865614 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:14.865617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:14.865619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:14.865823 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:14.866073 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:14.866076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:14.866078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:14.866156 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:14.866564 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:55:14.866567 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:55:14.866662 (+ 95us) client_negotiation.cc:769] Sending connection context
0810 03:55:14.866794 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 03:55:14.866943 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260810 03:55:14.867168 3883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.857718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44396 (local address 127.27.153.125:43115)
0810 03:55:14.857864 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:14.857869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:14.857885 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:55:14.857955 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:14.857958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:14.858009 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:14.858087 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:14.858630 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.859399 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.860361 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.860608 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.860702 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.863948 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:14.863952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:14.863954 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:14.863976 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:14.865516 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.865911 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.865914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.865916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.865992 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.866240 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.866243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.866245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.866472 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:14.866559 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:14.866860 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 03:55:14.866991 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":51}
I20260810 03:55:14.867770 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.859634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:34483 (local address 127.0.0.1:41144)
0810 03:55:14.859736 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.859749 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:55:14.859817 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:55:14.860116 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:55:14.860119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:55:14.860299 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:55:14.860421 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:14.860428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.861346 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.861350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:14.861904 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:14.861913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.862523 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.862527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:14.862626 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.863012 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:55:14.863026 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:55:14.864548 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:55:14.866210 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:14.866213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:14.866215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:14.866435 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:14.866737 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:14.866740 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:14.866742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:14.866831 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:14.867253 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:55:14.867256 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:55:14.867348 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 03:55:14.867422 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 03:55:14.867575 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":4}
I20260810 03:55:14.867933 4001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:55:14.868366 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.859605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41144 (local address 127.27.153.124:34483)
0810 03:55:14.859918 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:14.859921 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:14.859930 (+ 9us) server_negotiation.cc:407] Connection header received
0810 03:55:14.859955 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:14.859956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:14.859998 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:14.860528 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:14.860611 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.861238 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.862016 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.862384 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.862496 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.864691 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:14.864694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:14.864697 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:14.864719 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:14.866099 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.866550 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.866553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.866554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.866635 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.866939 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.866942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.866944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.867141 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:14.868025 (+ 884us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:14.868115 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 03:55:14.868229 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 03:55:14.868903 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.858091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39932 (local address 127.27.153.126:33399)
0810 03:55:14.858196 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:14.858200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:14.858283 (+ 83us) server_negotiation.cc:407] Connection header received
0810 03:55:14.858358 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:14.858361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:14.858408 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:14.858491 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:14.859157 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.859936 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.860996 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.861308 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.861408 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.864274 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:14.864279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:14.864281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:14.864312 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:14.865975 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.866530 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.866534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.866536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.866628 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.866935 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.866938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.866940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.867162 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:14.868244 (+ 1082us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:14.868336 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 03:55:14.868743 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 03:55:14.868974 3749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } attempt: 2
I20260810 03:55:14.873333 3870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7627b3decad46bb8be3dcf42dc951c5: Bootstrap starting.
Aug 10 03:55:14 dist-test-slave-pww0 krb5kdc[3618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334114, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.124@KRBTEST.COM for kudu/127.27.153.124@KRBTEST.COM
Aug 10 03:55:14 dist-test-slave-pww0 krb5kdc[3618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334114, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for kudu/127.27.153.124@KRBTEST.COM
Aug 10 03:55:14 dist-test-slave-pww0 krb5kdc[3618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.125@KRBTEST.COM
I20260810 03:55:14.876348 3870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7627b3decad46bb8be3dcf42dc951c5: Neither blocks nor log segments found. Creating new log.
I20260810 03:55:14.877367 3870 log.cc:824] T 00000000000000000000000000000000 P d7627b3decad46bb8be3dcf42dc951c5: Log is configured to *not* fsync() on all Append() calls
I20260810 03:55:14.879827 3870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7627b3decad46bb8be3dcf42dc951c5: No bootstrap required, opened a new log
I20260810 03:55:14.879882 3756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.869449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:34483 (local address 127.0.0.1:41172)
0810 03:55:14.869587 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.869602 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:55:14.869674 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:55:14.870848 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:55:14.870851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:55:14.871019 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:55:14.871115 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:14.871121 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.872153 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.872157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:14.872893 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:14.872906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.873527 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.873531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:14.873614 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.874064 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:55:14.874079 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:55:14.875613 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:55:14.878130 (+ 2517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:14.878134 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:14.878136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:14.878385 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:14.878703 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:14.878707 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:14.878709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:14.878804 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:14.879296 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:55:14.879299 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:55:14.879410 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 03:55:14.879541 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 03:55:14.879712 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 03:55:14.879899 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.868462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:34483 (local address 127.0.0.1:41156)
0810 03:55:14.869221 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:14.869236 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:55:14.869321 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:55:14.869785 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:55:14.869788 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:55:14.870084 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:55:14.870184 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:14.870190 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.871093 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.871096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:14.871927 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:14.871938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.872508 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.872511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:14.872583 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.873504 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:55:14.873518 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:55:14.875607 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:55:14.877738 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:14.877741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:14.877743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:14.878642 (+ 899us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:14.878876 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:14.878878 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:14.878881 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:14.878955 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:14.879387 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:55:14.879390 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:55:14.879500 (+ 110us) client_negotiation.cc:769] Sending connection context
0810 03:55:14.879599 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 03:55:14.879739 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696}
I20260810 03:55:14.880358 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.868511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41156 (local address 127.27.153.124:34483)
0810 03:55:14.868609 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:14.868613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:14.869535 (+ 922us) server_negotiation.cc:407] Connection header received
0810 03:55:14.869575 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:14.869578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:14.869622 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:14.869688 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:14.870290 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.870980 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.872034 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.872326 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.872408 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.875751 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:14.875755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:14.875758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:14.875780 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:14.877628 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.878729 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.878731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.878733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.878797 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.879044 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.879046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.879048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.879259 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:14.879352 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:14.880112 (+ 760us) server_negotiation.cc:299] Negotiation successful
0810 03:55:14.880228 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 03:55:14.880524 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.869762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41172 (local address 127.27.153.124:34483)
0810 03:55:14.870617 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:14.870626 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:14.870639 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:55:14.870668 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:14.870671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:14.870706 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:14.871228 (+ 522us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:14.871313 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.872026 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.873045 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.873393 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.873815 (+ 422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.875782 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:14.875785 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:14.875788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:14.875813 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:14.877993 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.878490 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.878493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.878495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.878585 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.878904 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.878908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.878910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.879133 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:14.879241 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:14.880188 (+ 947us) server_negotiation.cc:299] Negotiation successful
0810 03:55:14.880314 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":794,"thread_start_us":120,"threads_started":1}
I20260810 03:55:14.881230 3883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.870319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44412 (local address 127.27.153.125:43115)
0810 03:55:14.870433 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:14.870437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:14.870451 (+ 14us) server_negotiation.cc:407] Connection header received
0810 03:55:14.870487 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:14.870489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:14.870529 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:14.870595 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:14.871398 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.872133 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.873057 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.873287 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.873372 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.876636 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:14.876641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:14.876644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:14.876677 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:14.878211 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.878691 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.878695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.878697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.878783 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.879067 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.879070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.879072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.879259 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:14.879488 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:14.880388 (+ 900us) server_negotiation.cc:299] Negotiation successful
0810 03:55:14.881100 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 03:55:14.883955 3749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1: Bootstrap starting.
I20260810 03:55:14.883908 3870 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7627b3decad46bb8be3dcf42dc951c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464fb81536324a02ad6318f54a7b7ee1" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33399 } } peers { permanent_uuid: "d7627b3decad46bb8be3dcf42dc951c5" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } } peers { permanent_uuid: "152b4fbeee314501b596bc0273795427" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } }
I20260810 03:55:14.884182 3870 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7627b3decad46bb8be3dcf42dc951c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7627b3decad46bb8be3dcf42dc951c5, State: Initialized, Role: FOLLOWER
I20260810 03:55:14.884541 3870 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7627b3decad46bb8be3dcf42dc951c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464fb81536324a02ad6318f54a7b7ee1" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33399 } } peers { permanent_uuid: "d7627b3decad46bb8be3dcf42dc951c5" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } } peers { permanent_uuid: "152b4fbeee314501b596bc0273795427" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } }
I20260810 03:55:14.885977 3870 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7627b3decad46bb8be3dcf42dc951c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:55:14.886284 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:55:14.886855 3749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1: Neither blocks nor log segments found. Creating new log.
I20260810 03:55:14.887836 3749 log.cc:824] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1: Log is configured to *not* fsync() on all Append() calls
I20260810 03:55:14.887923 3703 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "464fb81536324a02ad6318f54a7b7ee1"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39932
I20260810 03:55:14.890789 4001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 152b4fbeee314501b596bc0273795427: Bootstrap starting.
I20260810 03:55:14.893559 4001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 152b4fbeee314501b596bc0273795427: Neither blocks nor log segments found. Creating new log.
I20260810 03:55:14.894500 4001 log.cc:824] T 00000000000000000000000000000000 P 152b4fbeee314501b596bc0273795427: Log is configured to *not* fsync() on all Append() calls
I20260810 03:55:14.888196 4013 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7627b3decad46bb8be3dcf42dc951c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464fb81536324a02ad6318f54a7b7ee1" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33399 } } peers { permanent_uuid: "d7627b3decad46bb8be3dcf42dc951c5" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } } peers { permanent_uuid: "152b4fbeee314501b596bc0273795427" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } } }
I20260810 03:55:14.895215 4013 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7627b3decad46bb8be3dcf42dc951c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:55:14.897367 4001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 152b4fbeee314501b596bc0273795427: No bootstrap required, opened a new log
I20260810 03:55:14.896791 3767 master_runner.cc:186] Error getting master registration for 127.27.153.126:33399: OK, instance_id { permanent_uuid: "464fb81536324a02ad6318f54a7b7ee1" instance_seqno: 1786334114337355 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 03:55:14.900404 3767 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 03:55:14.901510 4001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 152b4fbeee314501b596bc0273795427 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464fb81536324a02ad6318f54a7b7ee1" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33399 } } peers { permanent_uuid: "d7627b3decad46bb8be3dcf42dc951c5" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } } peers { permanent_uuid: "152b4fbeee314501b596bc0273795427" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } }
I20260810 03:55:14.901783 4001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 152b4fbeee314501b596bc0273795427 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 152b4fbeee314501b596bc0273795427, State: Initialized, Role: FOLLOWER
I20260810 03:55:14.902148 4001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 152b4fbeee314501b596bc0273795427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464fb81536324a02ad6318f54a7b7ee1" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33399 } } peers { permanent_uuid: "d7627b3decad46bb8be3dcf42dc951c5" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } } peers { permanent_uuid: "152b4fbeee314501b596bc0273795427" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } }
I20260810 03:55:14.902714 4047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152b4fbeee314501b596bc0273795427 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464fb81536324a02ad6318f54a7b7ee1" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33399 } } peers { permanent_uuid: "d7627b3decad46bb8be3dcf42dc951c5" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } } peers { permanent_uuid: "152b4fbeee314501b596bc0273795427" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } } }
I20260810 03:55:14.902865 4047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152b4fbeee314501b596bc0273795427 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 03:55:14.890866 4033 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d7627b3decad46bb8be3dcf42dc951c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:55:14.903584 4033 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:55:14.905200 3749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1: No bootstrap required, opened a new log
I20260810 03:55:14.907259 4001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 152b4fbeee314501b596bc0273795427 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:55:14.908478 3749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464fb81536324a02ad6318f54a7b7ee1" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33399 } } peers { permanent_uuid: "d7627b3decad46bb8be3dcf42dc951c5" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } } peers { permanent_uuid: "152b4fbeee314501b596bc0273795427" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } }
I20260810 03:55:14.908732 3749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 464fb81536324a02ad6318f54a7b7ee1, State: Initialized, Role: FOLLOWER
I20260810 03:55:14.909082 3749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464fb81536324a02ad6318f54a7b7ee1" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33399 } } peers { permanent_uuid: "d7627b3decad46bb8be3dcf42dc951c5" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } } peers { permanent_uuid: "152b4fbeee314501b596bc0273795427" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } }
I20260810 03:55:14.909729 4074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464fb81536324a02ad6318f54a7b7ee1" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33399 } } peers { permanent_uuid: "d7627b3decad46bb8be3dcf42dc951c5" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } } peers { permanent_uuid: "152b4fbeee314501b596bc0273795427" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } } }
I20260810 03:55:14.909878 4074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:55:14.910641 3749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:55:14.910912 3703 raft_consensus.cc:482] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 03:55:14.911041 3703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:55:14.912475 3703 raft_consensus.cc:504] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464fb81536324a02ad6318f54a7b7ee1" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33399 } } peers { permanent_uuid: "d7627b3decad46bb8be3dcf42dc951c5" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } } peers { permanent_uuid: "152b4fbeee314501b596bc0273795427" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } }
I20260810 03:55:14.913498 3703 leader_election.cc:288] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1 [CANDIDATE]: Term 1 election: Requested vote from peers d7627b3decad46bb8be3dcf42dc951c5 (127.27.153.125:43115), 152b4fbeee314501b596bc0273795427 (127.27.153.124:34483)
W20260810 03:55:14.914256 4075 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 152b4fbeee314501b596bc0273795427: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 03:55:14.914201 3831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "464fb81536324a02ad6318f54a7b7ee1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d7627b3decad46bb8be3dcf42dc951c5"
W20260810 03:55:14.914372 4075 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:55:14.914472 3831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7627b3decad46bb8be3dcf42dc951c5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:55:14.916013 3831 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d7627b3decad46bb8be3dcf42dc951c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 464fb81536324a02ad6318f54a7b7ee1 in term 1.
I20260810 03:55:14.916530 3963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "464fb81536324a02ad6318f54a7b7ee1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "152b4fbeee314501b596bc0273795427"
I20260810 03:55:14.916785 3963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 152b4fbeee314501b596bc0273795427 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:55:14.918195 3963 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 152b4fbeee314501b596bc0273795427 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 464fb81536324a02ad6318f54a7b7ee1 in term 1.
I20260810 03:55:14.915547 3885 master_runner.cc:186] Error getting master registration for 127.27.153.126:33399: OK, instance_id { permanent_uuid: "464fb81536324a02ad6318f54a7b7ee1" instance_seqno: 1786334114337355 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 03:55:14.918743 3885 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 03:55:14.919163 3662 leader_election.cc:302] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 152b4fbeee314501b596bc0273795427, 464fb81536324a02ad6318f54a7b7ee1; no voters:
I20260810 03:55:14.919516 4074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:55:14.919771 4074 raft_consensus.cc:686] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1 [term 1 LEADER]: Becoming Leader. State: Replica: 464fb81536324a02ad6318f54a7b7ee1, State: Running, Role: LEADER
I20260810 03:55:14.920113 4074 consensus_queue.cc:237] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464fb81536324a02ad6318f54a7b7ee1" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33399 } } peers { permanent_uuid: "d7627b3decad46bb8be3dcf42dc951c5" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } } peers { permanent_uuid: "152b4fbeee314501b596bc0273795427" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } }
W20260810 03:55:14.925712 4101 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:55:14.925808 4101 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:55:14.931120 4102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 464fb81536324a02ad6318f54a7b7ee1. Latest consensus state: current_term: 1 leader_uuid: "464fb81536324a02ad6318f54a7b7ee1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464fb81536324a02ad6318f54a7b7ee1" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33399 } } peers { permanent_uuid: "d7627b3decad46bb8be3dcf42dc951c5" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } } peers { permanent_uuid: "152b4fbeee314501b596bc0273795427" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } } }
I20260810 03:55:14.931341 4102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1 [sys.catalog]: This master's current role is: LEADER
I20260810 03:55:14.942606 3758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.927034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39934 (local address 127.27.153.126:33399)
0810 03:55:14.927263 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:14.927268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:14.929073 (+ 1805us) server_negotiation.cc:407] Connection header received
0810 03:55:14.929113 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:14.929116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:14.929161 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:14.929232 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:14.932433 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.933149 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.934203 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.934540 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.934612 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.938641 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:14.938648 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:14.938652 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:14.938687 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:14.940576 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.941106 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.941110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.941111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.941211 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.941536 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.941540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.941542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.941820 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:14.941941 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:14.942289 (+ 348us) server_negotiation.cc:299] Negotiation successful
0810 03:55:14.942429 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260810 03:55:14.944870 4112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:55:14.947173 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.926296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41180 (local address 127.27.153.124:34483)
0810 03:55:14.926429 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:14.926434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:14.928981 (+ 2547us) server_negotiation.cc:407] Connection header received
0810 03:55:14.930072 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:14.930075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:14.930131 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:14.930181 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:14.933094 (+ 2913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.933901 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.935539 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.935899 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.935983 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.940104 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:14.940109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:14.940112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:14.940158 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:14.941758 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.944498 (+ 2740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.944502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.944503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.944613 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.945388 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.945392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.945394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.945631 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:14.945766 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:14.946840 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0810 03:55:14.946989 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 03:55:14.948282 4112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:55:14.949260 3883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:14.927140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44426 (local address 127.27.153.125:43115)
0810 03:55:14.927353 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:14.927357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:14.928135 (+ 778us) server_negotiation.cc:407] Connection header received
0810 03:55:14.928594 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:14.928597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:14.928646 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:14.928695 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:14.931154 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.931895 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.934929 (+ 3034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:14.935246 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:14.935365 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:14.942292 (+ 6927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:14.942297 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:14.942300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:14.942331 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:14.943956 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.944951 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.944955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.944957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.945059 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:14.946086 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:14.946090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:14.946092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:14.946336 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:14.946456 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:14.948962 (+ 2506us) server_negotiation.cc:299] Negotiation successful
0810 03:55:14.949095 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260810 03:55:14.960143 3963 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 152b4fbeee314501b596bc0273795427 [term 1 FOLLOWER]: Refusing update from remote peer 464fb81536324a02ad6318f54a7b7ee1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:55:14.961094 3831 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d7627b3decad46bb8be3dcf42dc951c5 [term 1 FOLLOWER]: Refusing update from remote peer 464fb81536324a02ad6318f54a7b7ee1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:55:14.961951 4102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7627b3decad46bb8be3dcf42dc951c5" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:55:14.962950 4102 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "152b4fbeee314501b596bc0273795427" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:55:14.966938 4013 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7627b3decad46bb8be3dcf42dc951c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 464fb81536324a02ad6318f54a7b7ee1. Latest consensus state: current_term: 1 leader_uuid: "464fb81536324a02ad6318f54a7b7ee1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464fb81536324a02ad6318f54a7b7ee1" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33399 } } peers { permanent_uuid: "d7627b3decad46bb8be3dcf42dc951c5" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } } peers { permanent_uuid: "152b4fbeee314501b596bc0273795427" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } } }
I20260810 03:55:14.967092 4013 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7627b3decad46bb8be3dcf42dc951c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:55:14.970728 4047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152b4fbeee314501b596bc0273795427 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 464fb81536324a02ad6318f54a7b7ee1. Latest consensus state: current_term: 1 leader_uuid: "464fb81536324a02ad6318f54a7b7ee1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464fb81536324a02ad6318f54a7b7ee1" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33399 } } peers { permanent_uuid: "d7627b3decad46bb8be3dcf42dc951c5" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } } peers { permanent_uuid: "152b4fbeee314501b596bc0273795427" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } } }
I20260810 03:55:14.970873 4047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152b4fbeee314501b596bc0273795427 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:55:14.974522 4074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "464fb81536324a02ad6318f54a7b7ee1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464fb81536324a02ad6318f54a7b7ee1" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33399 } } peers { permanent_uuid: "d7627b3decad46bb8be3dcf42dc951c5" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } } peers { permanent_uuid: "152b4fbeee314501b596bc0273795427" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } } }
I20260810 03:55:14.974696 4074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1 [sys.catalog]: This master's current role is: LEADER
I20260810 03:55:14.974831 4114 mvcc.cc:205] Tried to move back new op lower bound from 7316824534868901888 to 7316824534716321792. Current Snapshot: MvccSnapshot[applied={T|T < 7316824534868901888}]
I20260810 03:55:14.975061 4047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 152b4fbeee314501b596bc0273795427 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "464fb81536324a02ad6318f54a7b7ee1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464fb81536324a02ad6318f54a7b7ee1" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33399 } } peers { permanent_uuid: "d7627b3decad46bb8be3dcf42dc951c5" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } } peers { permanent_uuid: "152b4fbeee314501b596bc0273795427" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } } }
I20260810 03:55:14.975199 4047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 152b4fbeee314501b596bc0273795427 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:55:14.977291 4112 catalog_manager.cc:1347] Generated new cluster ID: a351e80068f84cf39bf02253f39d6d2b
I20260810 03:55:14.977361 4112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:55:14.977541 4119 mvcc.cc:205] Tried to move back new op lower bound from 7316824534868901888 to 7316824534716321792. Current Snapshot: MvccSnapshot[applied={T|T < 7316824534868901888}]
I20260810 03:55:14.981714 4102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "464fb81536324a02ad6318f54a7b7ee1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464fb81536324a02ad6318f54a7b7ee1" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33399 } } peers { permanent_uuid: "d7627b3decad46bb8be3dcf42dc951c5" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } } peers { permanent_uuid: "152b4fbeee314501b596bc0273795427" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } } }
I20260810 03:55:14.981858 4102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1 [sys.catalog]: This master's current role is: LEADER
I20260810 03:55:14.983174 4013 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7627b3decad46bb8be3dcf42dc951c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "464fb81536324a02ad6318f54a7b7ee1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "464fb81536324a02ad6318f54a7b7ee1" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 33399 } } peers { permanent_uuid: "d7627b3decad46bb8be3dcf42dc951c5" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 43115 } } peers { permanent_uuid: "152b4fbeee314501b596bc0273795427" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 34483 } } }
I20260810 03:55:14.983321 4013 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7627b3decad46bb8be3dcf42dc951c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:55:15.022060 4112 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:55:15.023824 4112 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:55:15.037850 4112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 464fb81536324a02ad6318f54a7b7ee1: Generated new TSK 0
I20260810 03:55:15.038908 4112 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 03:55:15.080353 3883 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:15.077315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44442 (local address 127.27.153.125:43115)
0810 03:55:15.077467 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:15.077473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:15.079017 (+ 1544us) server_negotiation.cc:407] Connection header received
0810 03:55:15.079258 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:15.079262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:15.079326 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:15.079385 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:15.080195 (+ 810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44442: BlockingRecv error: recv got EOF from 127.0.0.1:44442 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 03:55:15.082255 3758 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:15.078302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39946 (local address 127.27.153.126:33399)
0810 03:55:15.078434 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:15.078440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:15.081168 (+ 2728us) server_negotiation.cc:407] Connection header received
0810 03:55:15.081371 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:15.081375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:15.081432 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:15.081488 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:15.082101 (+ 613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39946: BlockingRecv error: recv got EOF from 127.0.0.1:39946 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 03:55:15.086025 4011 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:15.083105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41194 (local address 127.27.153.124:34483)
0810 03:55:15.083302 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:15.083306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:15.083892 (+ 586us) server_negotiation.cc:407] Connection header received
0810 03:55:15.085167 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:15.085171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:15.085226 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:15.085282 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:15.085883 (+ 601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41194: BlockingRecv error: recv got EOF from 127.0.0.1:41194 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260810 03:55:15.091043 3883 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:15.089313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44444 (local address 127.27.153.125:43115)
0810 03:55:15.089427 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:15.089431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:15.089953 (+ 522us) server_negotiation.cc:407] Connection header received
0810 03:55:15.090172 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:15.090176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:15.090261 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:15.090340 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:15.090929 (+ 589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44444: BlockingRecv error: recv got EOF from 127.0.0.1:44444 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 03:55:15.096704 3883 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:15.094965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44456 (local address 127.27.153.125:43115)
0810 03:55:15.095077 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:15.095082 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:15.095695 (+ 613us) server_negotiation.cc:407] Connection header received
0810 03:55:15.095920 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:15.095923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:15.095983 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:15.096031 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:15.096590 (+ 559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44456: BlockingRecv error: recv got EOF from 127.0.0.1:44456 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 03:55:15.102698 3883 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:15.101470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44470 (local address 127.27.153.125:43115)
0810 03:55:15.101849 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:15.101854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:15.101872 (+ 18us) server_negotiation.cc:407] Connection header received
0810 03:55:15.101928 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:15.101931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:15.101986 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:15.102084 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:15.102582 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44470: BlockingRecv error: recv got EOF from 127.0.0.1:44470 (error 108)
Metrics: {"server-negotiator.queue_time_us":300}
W20260810 03:55:15.110191 3883 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:15.108908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44482 (local address 127.27.153.125:43115)
0810 03:55:15.109068 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:15.109073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:15.109241 (+ 168us) server_negotiation.cc:407] Connection header received
0810 03:55:15.109400 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:15.109404 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:15.109482 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:15.109587 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:15.110061 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44482: BlockingRecv error: recv got EOF from 127.0.0.1:44482 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260810 03:55:15.121912 3883 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:15.120528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44496 (local address 127.27.153.125:43115)
0810 03:55:15.120659 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:15.120663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:15.120992 (+ 329us) server_negotiation.cc:407] Connection header received
0810 03:55:15.121135 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:15.121139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:15.121207 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:15.121308 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:15.121784 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44496: BlockingRecv error: recv got EOF from 127.0.0.1:44496 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 03:55:15.141417 3883 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:15.140148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44504 (local address 127.27.153.125:43115)
0810 03:55:15.140307 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:15.140313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:15.140500 (+ 187us) server_negotiation.cc:407] Connection header received
0810 03:55:15.140650 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:15.140652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:15.140707 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:15.140790 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:15.141274 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44504: BlockingRecv error: recv got EOF from 127.0.0.1:44504 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 03:55:15.177016 3883 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:15.175755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44518 (local address 127.27.153.125:43115)
0810 03:55:15.175908 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:15.175913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:15.176089 (+ 176us) server_negotiation.cc:407] Connection header received
0810 03:55:15.176241 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:15.176244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:15.176309 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:15.176383 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:15.176884 (+ 501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44518: BlockingRecv error: recv got EOF from 127.0.0.1:44518 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 03:55:15.244635 3883 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:15.243383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44520 (local address 127.27.153.125:43115)
0810 03:55:15.243561 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:15.243567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:15.243761 (+ 194us) server_negotiation.cc:407] Connection header received
0810 03:55:15.243859 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:15.243862 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:15.243918 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:15.243995 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:15.244504 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44520: BlockingRecv error: recv got EOF from 127.0.0.1:44520 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260810 03:55:15.376390 3883 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:15.375307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44532 (local address 127.27.153.125:43115)
0810 03:55:15.375524 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:15.375529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:15.375592 (+ 63us) server_negotiation.cc:407] Connection header received
0810 03:55:15.375728 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:15.375730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:15.375781 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:15.375855 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:15.376258 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44532: BlockingRecv error: recv got EOF from 127.0.0.1:44532 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260810 03:55:15.636070 3883 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:15.634845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44546 (local address 127.27.153.125:43115)
0810 03:55:15.635032 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:15.635036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:15.635052 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:55:15.635204 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:15.635207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:15.635256 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:15.635330 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:15.635942 (+ 612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44546: BlockingRecv error: recv got EOF from 127.0.0.1:44546 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260810 03:55:15.904994 4033 catalog_manager.cc:1259] Loaded cluster ID: a351e80068f84cf39bf02253f39d6d2b
I20260810 03:55:15.905094 4033 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d7627b3decad46bb8be3dcf42dc951c5: loading cluster ID for follower catalog manager: success
I20260810 03:55:15.906946 4033 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d7627b3decad46bb8be3dcf42dc951c5: acquiring CA information for follower catalog manager: success
I20260810 03:55:15.908388 4033 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d7627b3decad46bb8be3dcf42dc951c5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:55:15.915783 4075 catalog_manager.cc:1259] Loaded cluster ID: a351e80068f84cf39bf02253f39d6d2b
I20260810 03:55:15.915868 4075 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 152b4fbeee314501b596bc0273795427: loading cluster ID for follower catalog manager: success
I20260810 03:55:15.917611 4075 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 152b4fbeee314501b596bc0273795427: acquiring CA information for follower catalog manager: success
I20260810 03:55:15.918835 4075 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 152b4fbeee314501b596bc0273795427: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:55:16.154836 4230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:16.150418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44548 (local address 127.27.153.125:43115)
0810 03:55:16.150687 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:16.150692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:16.150708 (+ 16us) server_negotiation.cc:407] Connection header received
0810 03:55:16.150892 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:16.150895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:16.151035 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:16.151146 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:55:16.151515 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:16.152117 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:16.153125 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:16.153363 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:16.153449 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:16.153857 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:55:16.154349 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:55:16.154453 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:16.154603 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 03:55:16.154694 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":73,"threads_started":1}
I20260810 03:55:16.162122 4244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:16.158221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41208 (local address 127.27.153.124:34483)
0810 03:55:16.158430 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:16.158433 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:16.158446 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:55:16.158561 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:16.158563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:16.158690 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:16.158790 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:55:16.159088 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:16.159687 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:16.160568 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:16.160830 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:16.160911 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:16.161240 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:55:16.161733 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:55:16.161838 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:16.161917 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 03:55:16.161976 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":65,"threads_started":1}
I20260810 03:55:16.169513 4230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:16.165562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44562 (local address 127.27.153.125:43115)
0810 03:55:16.165873 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:16.165878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:16.166026 (+ 148us) server_negotiation.cc:407] Connection header received
0810 03:55:16.166143 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:16.166146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:16.166222 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:16.166299 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:55:16.166660 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:16.167209 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:16.168340 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:16.168673 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:16.168768 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:16.168993 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:55:16.169083 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:55:16.169144 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:16.169302 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 03:55:16.169369 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221}
I20260810 03:55:16.169778 4244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:16.165666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41216 (local address 127.27.153.124:34483)
0810 03:55:16.165793 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:16.165798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:16.166284 (+ 486us) server_negotiation.cc:407] Connection header received
0810 03:55:16.166375 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:16.166377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:16.166428 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:16.166491 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:55:16.166829 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:16.167619 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:16.168660 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:16.168932 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:16.169030 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:16.169243 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:55:16.169320 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:55:16.169379 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:16.169599 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 03:55:16.169651 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 03:55:16.170473 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:16.165706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39954 (local address 127.27.153.126:33399)
0810 03:55:16.166005 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:16.166010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:16.166023 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:55:16.166188 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:16.166191 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:16.166360 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:16.166483 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:55:16.166748 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:16.167616 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:16.168625 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:16.168979 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:16.169074 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:16.169471 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:55:16.170086 (+ 615us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:55:16.170172 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:16.170260 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 03:55:16.170334 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":78,"threads_started":1}
I20260810 03:55:16.173575 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 3639
I20260810 03:55:16.197898 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 3767
I20260810 03:55:16.218532 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 3885
2026-08-10T03:55:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3407 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 03:55:16 dist-test-slave-pww0 krb5kdc[4265](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 03:55:16 dist-test-slave-pww0 krb5kdc[4265](info): set up 2 sockets
Aug 10 03:55:16 dist-test-slave-pww0 krb5kdc[4265](info): commencing operation
krb5kdc: starting...
W20260810 03:55:17.425225 28261 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/test-user.keytab.
Aug 10 03:55:17 dist-test-slave-pww0 krb5kdc[4265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T03:55:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T03:55:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.153.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
Entry for principal kudu/127.27.153.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab.
I20260810 03:55:17.558548 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:42855
--webserver_interface=127.27.153.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:42855,127.27.153.125:45685,127.27.153.124:42387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 03:55:17.673791 4290 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:55:17.674181 4290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:55:17.674227 4290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:55:17.674295 4290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 03:55:17.678675 4290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:55:17.678762 4290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:55:17.678791 4290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:55:17.678813 4290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:55:17.678834 4290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:55:17.684531 4290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:42855,127.27.153.125:45685,127.27.153.124:42387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.126.keytab
--principal=kudu/127.27.153.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.126:42855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.153.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:55:17.685738 4290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:55:17.686839 4290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:55:17.695598 4300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:17.695601 4297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:17.695644 4296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:55:17.695964 4290 server_base.cc:1034] running on GCE node
I20260810 03:55:17.696538 4290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:55:17.697719 4290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:55:17.698896 4290 hybrid_clock.cc:648] HybridClock initialized: now 1786334117698879 us; error 29 us; skew 500 ppm
Aug 10 03:55:17 dist-test-slave-pww0 krb5kdc[4265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334117, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:55:17.705443 4290 init.cc:377] Logged in from keytab as kudu/127.27.153.126@KRBTEST.COM (short username kudu)
I20260810 03:55:17.706537 4290 webserver.cc:460] Webserver started at http://127.27.153.126:43213/ using document root <none> and password file <none>
I20260810 03:55:17.707134 4290 fs_manager.cc:359] Metadata directory not provided
I20260810 03:55:17.707209 4290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:55:17.707423 4290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:55:17.709582 4290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-0/data/instance:
uuid: "d4e6de7ee2f74ee39569b2e6039c00d9"
format_stamp: "Formatted at 2026-08-10 03:55:17 on dist-test-slave-pww0"
server_key: "a6820ede387b1dce1b5612771b3a018f"
server_key_iv: "067a2e6ed238f0fec591d467379c3536"
server_key_version: "encryptionkey@0"
I20260810 03:55:17.710160 4290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-0/wal/instance:
uuid: "d4e6de7ee2f74ee39569b2e6039c00d9"
format_stamp: "Formatted at 2026-08-10 03:55:17 on dist-test-slave-pww0"
server_key: "a6820ede387b1dce1b5612771b3a018f"
server_key_iv: "067a2e6ed238f0fec591d467379c3536"
server_key_version: "encryptionkey@0"
I20260810 03:55:17.714082 4290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260810 03:55:17.716684 4308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:55:17.717950 4290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 03:55:17.718077 4290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-0/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-0/wal
uuid: "d4e6de7ee2f74ee39569b2e6039c00d9"
format_stamp: "Formatted at 2026-08-10 03:55:17 on dist-test-slave-pww0"
server_key: "a6820ede387b1dce1b5612771b3a018f"
server_key_iv: "067a2e6ed238f0fec591d467379c3536"
server_key_version: "encryptionkey@0"
I20260810 03:55:17.718182 4290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:55:17.733733 4290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:55:17.736837 4290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:55:17.737033 4290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:55:17.744323 4290 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.126:42855
I20260810 03:55:17.744602 4396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.126:42855 every 8 connection(s)
I20260810 03:55:17.745365 4290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-0/data/info.pb
I20260810 03:55:17.746572 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 4290
I20260810 03:55:17.746699 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-0/wal/instance
I20260810 03:55:17.747028 28261 external_mini_cluster.cc:1428] Setting key 8ca824f4125137e4317c385d31102ba5
I20260810 03:55:17.748993 4398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:55:17.753343 4398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42855 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.153.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.125@KRBTEST.COM" created.
Aug 10 03:55:17 dist-test-slave-pww0 krb5kdc[4265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334117, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:55:17.768973 4405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:17.756902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38216 (local address 127.27.153.126:42855)
0810 03:55:17.757211 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:17.757223 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:17.757286 (+ 63us) server_negotiation.cc:407] Connection header received
0810 03:55:17.758009 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:17.758015 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:17.758252 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:17.758432 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:17.759317 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:17.760231 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:17.761223 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:17.761604 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:17.761748 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:17.764145 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:17.764152 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:17.764166 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:17.764187 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:17.765745 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:17.766191 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:17.766196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:17.766197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:17.766312 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:17.766625 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:17.766629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:17.766631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:17.767036 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:17.767167 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:17.767602 (+ 435us) server_negotiation.cc:299] Negotiation successful
0810 03:55:17.767848 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":204,"thread_start_us":90,"threads_started":1}
I20260810 03:55:17.768970 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:17.756044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42855 (local address 127.0.0.1:38216)
0810 03:55:17.756590 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:17.756638 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 03:55:17.757659 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:55:17.758449 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:55:17.758455 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:55:17.758911 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:55:17.759159 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:17.759173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:17.760357 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:17.760360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:17.761107 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:17.761117 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:17.761749 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:17.761753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:17.761917 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:17.762479 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:55:17.762495 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:55:17.764002 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:55:17.765850 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:17.765859 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:17.765871 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:17.766089 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:17.766396 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:17.766399 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:17.766401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:17.766515 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:17.767137 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:55:17.767144 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:55:17.767324 (+ 180us) client_negotiation.cc:769] Sending connection context
0810 03:55:17.767503 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 03:55:17.767909 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":116,"thread_start_us":154,"threads_started":1}
I20260810 03:55:17.771617 4398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:55:17.772460 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:17.772162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:45685 (local address 127.0.0.1:47986)
0810 03:55:17.772303 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:17.772387 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.125:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260810 03:55:17.772717 4314 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.153.125:45685: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.125.keytab.
Entry for principal kudu/127.27.153.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.125.keytab.
I20260810 03:55:17.773618 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:17.773388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:45685 (local address 127.0.0.1:48002)
0810 03:55:17.773499 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:17.773556 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.125:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 03:55:17.773792 4398 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.125:45685: Network error: Client connection negotiation failed: client connection to 127.27.153.125:45685: connect: Connection refused (error 111)
I20260810 03:55:17.774302 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.125:45685
--webserver_interface=127.27.153.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.125.keytab
--principal=kudu/127.27.153.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:42855,127.27.153.125:45685,127.27.153.124:42387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:55:17.823035 4398 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } attempt: 1
I20260810 03:55:17.823940 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:17.823643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:45685 (local address 127.0.0.1:48016)
0810 03:55:17.823794 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:17.823859 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.125:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 03:55:17.824654 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:17.824426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:45685 (local address 127.0.0.1:48022)
0810 03:55:17.824535 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:17.824582 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.125:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 03:55:17.824807 4398 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.125:45685: Network error: Client connection negotiation failed: client connection to 127.27.153.125:45685: connect: Connection refused (error 111)
W20260810 03:55:17.892021 4409 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:55:17.892378 4409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:55:17.892429 4409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:55:17.892495 4409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 03:55:17.892992 4398 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } attempt: 2
I20260810 03:55:17.893952 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:17.893581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:45685 (local address 127.0.0.1:48036)
0810 03:55:17.893775 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:17.893852 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.125:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 03:55:17.894893 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:17.894646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:45685 (local address 127.0.0.1:48042)
0810 03:55:17.894762 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:17.894819 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.125:45685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 03:55:17.895061 4398 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.125:45685: Network error: Client connection negotiation failed: client connection to 127.27.153.125:45685: connect: Connection refused (error 111)
W20260810 03:55:17.897172 4409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:55:17.897233 4409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:55:17.897251 4409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:55:17.897269 4409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:55:17.897305 4409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:55:17.903015 4409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:42855,127.27.153.125:45685,127.27.153.124:42387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.125.keytab
--principal=kudu/127.27.153.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.125:45685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.153.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:55:17.904202 4409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:55:17.905261 4409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:55:17.913805 4422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:17.913790 4418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:17.913841 4424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:55:17.914193 4409 server_base.cc:1034] running on GCE node
I20260810 03:55:17.914788 4409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:55:17.915989 4409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:55:17.917155 4409 hybrid_clock.cc:648] HybridClock initialized: now 1786334117917129 us; error 45 us; skew 500 ppm
Aug 10 03:55:17 dist-test-slave-pww0 krb5kdc[4265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334117, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:55:17.919988 4409 init.cc:377] Logged in from keytab as kudu/127.27.153.125@KRBTEST.COM (short username kudu)
I20260810 03:55:17.920917 4409 webserver.cc:460] Webserver started at http://127.27.153.125:42881/ using document root <none> and password file <none>
I20260810 03:55:17.921509 4409 fs_manager.cc:359] Metadata directory not provided
I20260810 03:55:17.921573 4409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:55:17.921783 4409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:55:17.924010 4409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-1/data/instance:
uuid: "701751cd74534d82837beba49375b4c4"
format_stamp: "Formatted at 2026-08-10 03:55:17 on dist-test-slave-pww0"
server_key: "f8acccca99e7c4e7a916a4d87f536ca1"
server_key_iv: "2684bfeb1b85f74ecbc7626e77841130"
server_key_version: "encryptionkey@0"
I20260810 03:55:17.924594 4409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-1/wal/instance:
uuid: "701751cd74534d82837beba49375b4c4"
format_stamp: "Formatted at 2026-08-10 03:55:17 on dist-test-slave-pww0"
server_key: "f8acccca99e7c4e7a916a4d87f536ca1"
server_key_iv: "2684bfeb1b85f74ecbc7626e77841130"
server_key_version: "encryptionkey@0"
I20260810 03:55:17.928409 4409 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 03:55:17.931046 4436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:55:17.932437 4409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 03:55:17.932560 4409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-1/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-1/wal
uuid: "701751cd74534d82837beba49375b4c4"
format_stamp: "Formatted at 2026-08-10 03:55:17 on dist-test-slave-pww0"
server_key: "f8acccca99e7c4e7a916a4d87f536ca1"
server_key_iv: "2684bfeb1b85f74ecbc7626e77841130"
server_key_version: "encryptionkey@0"
I20260810 03:55:17.932680 4409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:55:17.954586 4409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:55:17.958184 4409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:55:17.958384 4409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:55:17.965513 4409 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.125:45685
I20260810 03:55:17.965807 4525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.125:45685 every 8 connection(s)
I20260810 03:55:17.966508 4409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-1/data/info.pb
I20260810 03:55:17.969432 4526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:55:17.972257 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 4409
I20260810 03:55:17.972378 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-1/wal/instance
I20260810 03:55:17.972627 28261 external_mini_cluster.cc:1428] Setting key d286e6e0b3cdeecd833c8ef25579468b
I20260810 03:55:17.973495 4526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42855 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.153.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.153.124@KRBTEST.COM" created.
Aug 10 03:55:17 dist-test-slave-pww0 krb5kdc[4265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334117, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.125@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:55:17.986209 4398 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } attempt: 3
I20260810 03:55:17.990456 4405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:17.975587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38226 (local address 127.27.153.126:42855)
0810 03:55:17.975728 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:17.975732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:17.976861 (+ 1129us) server_negotiation.cc:407] Connection header received
0810 03:55:17.977966 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:17.977970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:17.978039 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:17.978154 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:17.979415 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:17.980273 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:17.981611 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:17.981989 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:17.982095 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:17.985595 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:17.985599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:17.985602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:17.985641 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:17.987312 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:17.988007 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:17.988011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:17.988013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:17.988119 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:17.988549 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:17.988552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:17.988554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:17.988740 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:17.988827 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:17.990151 (+ 1324us) server_negotiation.cc:299] Negotiation successful
0810 03:55:17.990310 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:55:17.991011 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:17.976113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42855 (local address 127.0.0.1:38226)
0810 03:55:17.976668 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:17.976731 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 03:55:17.977691 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:55:17.978323 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:55:17.978331 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:55:17.978714 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:55:17.979222 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:17.979247 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:17.980457 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:17.980460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:17.981456 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:17.981468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:17.982164 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:17.982167 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:17.982345 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:17.983016 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:55:17.983034 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:55:17.985354 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:55:17.987443 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:17.987450 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:17.987492 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:17.987865 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:17.988234 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:17.988237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:17.988239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:17.988434 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:17.988847 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:55:17.988854 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:55:17.989121 (+ 267us) client_negotiation.cc:769] Sending connection context
0810 03:55:17.989286 (+ 165us) client_negotiation.cc:240] Negotiation successful
0810 03:55:17.989661 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":136,"threads_started":1}
I20260810 03:55:17.992846 4526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 03:55:17 dist-test-slave-pww0 krb5kdc[4265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334117, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for kudu/127.27.153.125@KRBTEST.COM
I20260810 03:55:17.997362 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:17.986900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:45685 (local address 127.0.0.1:48050)
0810 03:55:17.987120 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:17.987135 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:55:17.987222 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:55:17.988065 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:55:17.988068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:55:17.988239 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:55:17.988390 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:17.988398 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:17.989749 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:17.989753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:17.990590 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:17.990603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:17.991483 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:17.991488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:17.991602 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:17.992063 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:55:17.992078 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:55:17.993407 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:55:17.995766 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:17.995770 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:17.995773 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:17.995970 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:17.996265 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:17.996268 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:17.996271 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:17.996352 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:17.996838 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:55:17.996841 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:55:17.996958 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 03:55:17.997084 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 03:55:17.997232 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260810 03:55:17.997443 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:17.987074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48050 (local address 127.27.153.125:45685)
0810 03:55:17.987536 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:17.987549 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:17.987576 (+ 27us) server_negotiation.cc:407] Connection header received
0810 03:55:17.987620 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:17.987627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:17.987889 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:17.988069 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:17.988530 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:17.989595 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:17.990751 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:17.991303 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:17.991836 (+ 533us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:17.993769 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:17.993779 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:17.993793 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:17.993812 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:17.995656 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:17.996078 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:17.996085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:17.996086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:17.996175 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:17.996434 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:17.996437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:17.996439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:17.996738 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:17.996877 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:17.997152 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 03:55:17.997287 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":146,"threads_started":1}
Aug 10 03:55:17 dist-test-slave-pww0 krb5kdc[4265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334117, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.125@KRBTEST.COM for kudu/127.27.153.125@KRBTEST.COM
I20260810 03:55:17.999399 4398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:55:18.000262 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.000001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:42387 (local address 127.0.0.1:60276)
0810 03:55:18.000127 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:18.000184 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 03:55:18.001199 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.000945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:42387 (local address 127.0.0.1:60286)
0810 03:55:18.001087 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:18.001136 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260810 03:55:18.001328 4398 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.124:42387: Network error: Client connection negotiation failed: client connection to 127.27.153.124:42387: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.153.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.124.keytab.
Entry for principal kudu/127.27.153.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.124.keytab.
I20260810 03:55:18.002780 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:17.993422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:45685 (local address 127.0.0.1:48060)
0810 03:55:17.993574 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:17.993587 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:55:17.993657 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:55:17.994122 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:55:17.994126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:55:17.994291 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:55:17.994399 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:17.994405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:17.995241 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:17.995245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:17.995899 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:17.995908 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:17.996349 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:17.996353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:17.996440 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:17.997561 (+ 1121us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:55:17.997575 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:55:17.999396 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:55:18.001428 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:18.001432 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:18.001434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:18.001665 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:18.001937 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:18.001939 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:18.001940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:18.002006 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:18.002343 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:55:18.002346 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:55:18.002431 (+ 85us) client_negotiation.cc:769] Sending connection context
0810 03:55:18.002510 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 03:55:18.002648 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260810 03:55:18.002923 4540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:17.993468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48060 (local address 127.27.153.125:45685)
0810 03:55:17.993881 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:17.993889 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:17.993904 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:55:17.993937 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:17.993940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:17.993987 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:17.994081 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:17.994503 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:17.995145 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:17.995997 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:17.996238 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:17.996369 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:17.999594 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:17.999598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:17.999600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:17.999625 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:18.001158 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.001772 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.001775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.001776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.001859 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.002067 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.002069 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.002070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.002258 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:18.002342 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:18.002545 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 03:55:18.002731 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":341,"thread_start_us":186,"threads_started":1}
I20260810 03:55:18.003675 4526 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:55:18.003911 28261 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
/tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.124:42387
--webserver_interface=127.27.153.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.124.keytab
--principal=kudu/127.27.153.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.153.84:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.153.126:42855,127.27.153.125:45685,127.27.153.124:42387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 03:55:18.004477 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.004210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:42387 (local address 127.0.0.1:60294)
0810 03:55:18.004325 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:18.004414 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 03:55:18.004717 4441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.153.124:42387: connect: Connection refused (error 111)
I20260810 03:55:18.005594 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.005365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:42387 (local address 127.0.0.1:60300)
0810 03:55:18.005470 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:18.005523 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 03:55:18.005769 4526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.124:42387: Network error: Client connection negotiation failed: client connection to 127.27.153.124:42387: connect: Connection refused (error 111)
I20260810 03:55:18.032490 4398 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } attempt: 1
I20260810 03:55:18.033440 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.033082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:42387 (local address 127.0.0.1:60306)
0810 03:55:18.033261 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:18.033339 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 03:55:18.034225 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.033986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:42387 (local address 127.0.0.1:60316)
0810 03:55:18.034096 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:18.034155 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 03:55:18.034408 4398 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.124:42387: Network error: Client connection negotiation failed: client connection to 127.27.153.124:42387: connect: Connection refused (error 111)
I20260810 03:55:18.055075 4526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } attempt: 1
I20260810 03:55:18.056018 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.055694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:42387 (local address 127.0.0.1:60320)
0810 03:55:18.055859 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:18.055928 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 03:55:18.056871 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.056619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:42387 (local address 127.0.0.1:60322)
0810 03:55:18.056752 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:18.056805 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 03:55:18.056996 4526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.124:42387: Network error: Client connection negotiation failed: client connection to 127.27.153.124:42387: connect: Connection refused (error 111)
I20260810 03:55:18.109591 4398 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } attempt: 2
I20260810 03:55:18.110596 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.110235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:42387 (local address 127.0.0.1:60328)
0810 03:55:18.110420 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:18.110496 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 03:55:18.111330 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.111098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:42387 (local address 127.0.0.1:60332)
0810 03:55:18.111201 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:18.111258 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 03:55:18.111476 4398 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.124:42387: Network error: Client connection negotiation failed: client connection to 127.27.153.124:42387: connect: Connection refused (error 111)
W20260810 03:55:18.121795 4544 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 03:55:18.122097 4544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 03:55:18.122123 4544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 03:55:18.122160 4544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 03:55:18.125133 4526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } attempt: 2
I20260810 03:55:18.126037 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.125697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:42387 (local address 127.0.0.1:60348)
0810 03:55:18.125869 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:18.125949 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260810 03:55:18.126541 4544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 03:55:18.126626 4544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 03:55:18.126657 4544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 03:55:18.126689 4544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 03:55:18.126719 4544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 03:55:18.126811 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.126525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:42387 (local address 127.0.0.1:60354)
0810 03:55:18.126674 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:18.126739 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.153.124:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 03:55:18.126972 4526 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.153.124:42387: Network error: Client connection negotiation failed: client connection to 127.27.153.124:42387: connect: Connection refused (error 111)
I20260810 03:55:18.132798 4544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.153.84:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.153.126:42855,127.27.153.125:45685,127.27.153.124:42387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/krb5kdc/kudu_127.27.153.124.keytab
--principal=kudu/127.27.153.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.153.124:42387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.153.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 02:54:48 UTC on ccycloud.quasar-oaausp.root.comops.site
build id 4576979
I20260810 03:55:18.134035 4544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 03:55:18.135146 4544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 03:55:18.144469 4554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:18.144469 4555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 03:55:18.144470 4558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 03:55:18.144898 4544 server_base.cc:1034] running on GCE node
I20260810 03:55:18.145547 4544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 03:55:18.146867 4544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 03:55:18.148142 4544 hybrid_clock.cc:648] HybridClock initialized: now 1786334118148122 us; error 43 us; skew 500 ppm
Aug 10 03:55:18 dist-test-slave-pww0 krb5kdc[4265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334118, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 03:55:18.151382 4544 init.cc:377] Logged in from keytab as kudu/127.27.153.124@KRBTEST.COM (short username kudu)
I20260810 03:55:18.152539 4544 webserver.cc:460] Webserver started at http://127.27.153.124:38191/ using document root <none> and password file <none>
I20260810 03:55:18.153160 4544 fs_manager.cc:359] Metadata directory not provided
I20260810 03:55:18.153235 4544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 03:55:18.153451 4544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 03:55:18.155800 4544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-2/data/instance:
uuid: "be6e19a2ca0f49ef9face5a36c093816"
format_stamp: "Formatted at 2026-08-10 03:55:18 on dist-test-slave-pww0"
server_key: "49e99b5e2ed1a35acdbc37b463531aa1"
server_key_iv: "235e3b5e80e93a2d60af932a241c0327"
server_key_version: "encryptionkey@0"
I20260810 03:55:18.156415 4544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-2/wal/instance:
uuid: "be6e19a2ca0f49ef9face5a36c093816"
format_stamp: "Formatted at 2026-08-10 03:55:18 on dist-test-slave-pww0"
server_key: "49e99b5e2ed1a35acdbc37b463531aa1"
server_key_iv: "235e3b5e80e93a2d60af932a241c0327"
server_key_version: "encryptionkey@0"
I20260810 03:55:18.160503 4544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 03:55:18.163254 4567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 03:55:18.164667 4544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 03:55:18.164820 4544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-2/data,/tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-2/wal
uuid: "be6e19a2ca0f49ef9face5a36c093816"
format_stamp: "Formatted at 2026-08-10 03:55:18 on dist-test-slave-pww0"
server_key: "49e99b5e2ed1a35acdbc37b463531aa1"
server_key_iv: "235e3b5e80e93a2d60af932a241c0327"
server_key_version: "encryptionkey@0"
I20260810 03:55:18.164954 4544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 03:55:18.184291 4544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 03:55:18.187426 4544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 03:55:18.187686 4544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 03:55:18.194700 4544 rpc_server.cc:307] RPC server started. Bound to: 127.27.153.124:42387
I20260810 03:55:18.195026 4656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.153.124:42387 every 8 connection(s)
I20260810 03:55:18.195756 4544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-2/data/info.pb
I20260810 03:55:18.198634 4657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 03:55:18.201994 28261 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu as pid 4544
I20260810 03:55:18.202121 28261 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM5UkAJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786333814837068-28261-0/minicluster-data/master-2/wal/instance
I20260810 03:55:18.202397 28261 external_mini_cluster.cc:1428] Setting key 63c3b17404fb8970e7961d9e4979308b
I20260810 03:55:18.202737 4657 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42855 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:55:18.210631 4398 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } attempt: 3
Aug 10 03:55:18 dist-test-slave-pww0 krb5kdc[4265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334118, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.124@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
Aug 10 03:55:18 dist-test-slave-pww0 krb5kdc[4265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.124@KRBTEST.COM
Aug 10 03:55:18 dist-test-slave-pww0 krb5kdc[4265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334117, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.126@KRBTEST.COM for kudu/127.27.153.124@KRBTEST.COM
I20260810 03:55:18.216284 4405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.204778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38236 (local address 127.27.153.126:42855)
0810 03:55:18.204925 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.204929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.205256 (+ 327us) server_negotiation.cc:407] Connection header received
0810 03:55:18.205667 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.205671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.205729 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.205825 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.206739 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.207255 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.208342 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.208582 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.208661 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:18.211518 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:18.211522 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:18.211526 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:18.211557 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:18.213314 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.213926 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.213929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.213931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.214027 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.214769 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.214773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.214774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.215024 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:18.215123 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:18.215972 (+ 849us) server_negotiation.cc:299] Negotiation successful
0810 03:55:18.216106 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 03:55:18.217746 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.204709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.126:42855 (local address 127.0.0.1:38236)
0810 03:55:18.205094 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:18.205148 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 03:55:18.205438 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:55:18.205889 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:55:18.205894 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:55:18.206284 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:55:18.206593 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:18.206606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.207357 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.207361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:18.208219 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:18.208230 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.208722 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.208726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:18.208933 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:18.209555 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:55:18.209572 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:55:18.211315 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:55:18.213433 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:18.213445 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:18.213459 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:18.213823 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:18.214451 (+ 628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:18.214454 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:18.214456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:18.214661 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:18.215542 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:55:18.215549 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:55:18.215691 (+ 142us) client_negotiation.cc:769] Sending connection context
0810 03:55:18.216506 (+ 815us) client_negotiation.cc:240] Negotiation successful
0810 03:55:18.216691 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":1,"spinlock_wait_cycles":42624,"thread_start_us":114,"threads_started":1}
I20260810 03:55:18.217746 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.203850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60360 (local address 127.27.153.124:42387)
0810 03:55:18.204270 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.204286 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.204319 (+ 33us) server_negotiation.cc:407] Connection header received
0810 03:55:18.205095 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.205119 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.205452 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.205698 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.206516 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.207537 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.208393 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.208900 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.208997 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:18.212237 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:18.212247 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:18.212263 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:18.212293 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:18.214224 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.214701 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.214708 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.214710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.214811 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.215118 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.215121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.215123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.215598 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:18.215814 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:18.216047 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 03:55:18.216642 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":278,"spinlock_wait_cycles":2944,"thread_start_us":117,"threads_started":1}
I20260810 03:55:18.219152 4526 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } attempt: 3
I20260810 03:55:18.219417 4657 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:55:18.221540 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.211079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:42387 (local address 127.0.0.1:60366)
0810 03:55:18.211320 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:18.211335 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:55:18.211421 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:55:18.211842 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:55:18.211845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:55:18.212016 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:55:18.212102 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:18.212108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.213005 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.213009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:18.213650 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:18.213661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.214158 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.214163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:18.214257 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:18.214700 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:55:18.214714 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:55:18.215979 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:55:18.218651 (+ 2672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:18.218654 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:18.218656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:18.218935 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:18.220384 (+ 1449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:18.220387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:18.220389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:18.220506 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:18.221011 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:55:18.221014 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:55:18.221124 (+ 110us) client_negotiation.cc:769] Sending connection context
0810 03:55:18.221228 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 03:55:18.221379 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260810 03:55:18.222121 4666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.211140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60366 (local address 127.27.153.124:42387)
0810 03:55:18.211626 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.211636 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.211646 (+ 10us) server_negotiation.cc:407] Connection header received
0810 03:55:18.211684 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.211687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.211724 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.211800 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.212213 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.212899 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.213762 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.214028 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.214123 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:18.216330 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:18.216334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:18.216337 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:18.216362 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:18.218507 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.219038 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.219042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.219044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.219136 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.220628 (+ 1492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.220632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.220633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.220917 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:18.221007 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:18.221775 (+ 768us) server_negotiation.cc:299] Negotiation successful
0810 03:55:18.221899 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":424,"thread_start_us":131,"threads_started":1}
Aug 10 03:55:18 dist-test-slave-pww0 krb5kdc[4265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334117, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.125@KRBTEST.COM for kudu/127.27.153.124@KRBTEST.COM
I20260810 03:55:18.225793 4398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9: Bootstrap starting.
Aug 10 03:55:18 dist-test-slave-pww0 krb5kdc[4265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334118, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.124@KRBTEST.COM for kudu/127.27.153.125@KRBTEST.COM
Aug 10 03:55:18 dist-test-slave-pww0 krb5kdc[4265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.126@KRBTEST.COM
I20260810 03:55:18.228940 4398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9: Neither blocks nor log segments found. Creating new log.
I20260810 03:55:18.229657 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.219657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:42387 (local address 127.0.0.1:60378)
0810 03:55:18.219861 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:18.219875 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 03:55:18.219962 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:55:18.220310 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:55:18.220313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:55:18.220485 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:55:18.220587 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:18.220593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.221891 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.221895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:18.222608 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:18.222619 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.223512 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.223516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:18.223600 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:18.224105 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:55:18.224121 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:55:18.225637 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:55:18.227509 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:18.227513 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:18.227516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:18.227765 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:18.228380 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:18.228383 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:18.228386 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:18.228489 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:18.229112 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:55:18.229116 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:55:18.229242 (+ 126us) client_negotiation.cc:769] Sending connection context
0810 03:55:18.229326 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 03:55:18.229483 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260810 03:55:18.229921 4398 log.cc:824] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9: Log is configured to *not* fsync() on all Append() calls
I20260810 03:55:18.230372 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.219682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60378 (local address 127.27.153.124:42387)
0810 03:55:18.219930 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.219934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.219949 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:55:18.220119 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.220122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.220178 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.220680 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.220755 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.221481 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.223014 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.223350 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.223826 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:18.225787 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:18.225791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:18.225793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:18.225819 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:18.227382 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.228202 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.228206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.228208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.228291 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.228766 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.228769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.228771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.228989 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:18.229982 (+ 993us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:18.230061 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 03:55:18.230190 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":171}
I20260810 03:55:18.232388 4540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.220295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48070 (local address 127.27.153.125:45685)
0810 03:55:18.220741 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.220745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.220758 (+ 13us) server_negotiation.cc:407] Connection header received
0810 03:55:18.220810 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.220813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.220856 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.220931 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.221819 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.222538 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.223676 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.224335 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.224412 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:18.228442 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:18.228446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:18.228449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:18.228475 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:18.230170 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.230907 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.230911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.230913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.231006 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.231315 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.231318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.231320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.231633 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:18.231702 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:18.232069 (+ 367us) server_negotiation.cc:299] Negotiation successful
0810 03:55:18.232201 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357}
I20260810 03:55:18.232649 4398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9: No bootstrap required, opened a new log
I20260810 03:55:18.233242 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.220014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.125:45685 (local address 127.0.0.1:48070)
0810 03:55:18.220192 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:18.220205 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 03:55:18.220273 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:55:18.221381 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:55:18.221384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:55:18.221557 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:55:18.221684 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:18.221692 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.222678 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.222681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:18.223528 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:18.223539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.224558 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.224562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:18.224679 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:18.225128 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:55:18.225142 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:55:18.228084 (+ 2942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:55:18.230285 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:18.230288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:18.230290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:18.230785 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:18.231124 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:18.231127 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:18.231129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:18.231216 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:18.231787 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:55:18.231790 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:55:18.231886 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 03:55:18.232930 (+ 1044us) client_negotiation.cc:240] Negotiation successful
0810 03:55:18.233073 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20260810 03:55:18.233548 4405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.221143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38246 (local address 127.27.153.126:42855)
0810 03:55:18.221256 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.221260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.221272 (+ 12us) server_negotiation.cc:407] Connection header received
0810 03:55:18.221337 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.221340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.221380 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.221443 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.222858 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.223626 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.224662 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.224962 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.225055 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:18.227902 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:18.227906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:18.227909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:18.227936 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:18.231152 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.231720 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.231724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.231725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.231814 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.232113 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.232116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.232118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.232366 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:18.232583 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:18.233278 (+ 695us) server_negotiation.cc:299] Negotiation successful
0810 03:55:18.233413 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 03:55:18.233804 4526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 701751cd74534d82837beba49375b4c4: Bootstrap starting.
I20260810 03:55:18.234632 4657 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } has no permanent_uuid. Determining permanent_uuid...
I20260810 03:55:18.237005 4398 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42855 } } peers { permanent_uuid: "701751cd74534d82837beba49375b4c4" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } } peers { permanent_uuid: "be6e19a2ca0f49ef9face5a36c093816" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } }
I20260810 03:55:18.237314 4526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 701751cd74534d82837beba49375b4c4: Neither blocks nor log segments found. Creating new log.
I20260810 03:55:18.237272 4398 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e6de7ee2f74ee39569b2e6039c00d9, State: Initialized, Role: FOLLOWER
I20260810 03:55:18.237694 4398 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42855 } } peers { permanent_uuid: "701751cd74534d82837beba49375b4c4" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } } peers { permanent_uuid: "be6e19a2ca0f49ef9face5a36c093816" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } }
I20260810 03:55:18.238243 4526 log.cc:824] T 00000000000000000000000000000000 P 701751cd74534d82837beba49375b4c4: Log is configured to *not* fsync() on all Append() calls
W20260810 03:55:18.238915 4316 tablet.cc:2369] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 03:55:18.239121 4398 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:55:18.240890 4673 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42855 } } peers { permanent_uuid: "701751cd74534d82837beba49375b4c4" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } } peers { permanent_uuid: "be6e19a2ca0f49ef9face5a36c093816" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } } }
I20260810 03:55:18.241053 4673 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9 [sys.catalog]: This master's current role is: FOLLOWER
Aug 10 03:55:18 dist-test-slave-pww0 krb5kdc[4265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.153.125@KRBTEST.COM
Aug 10 03:55:18 dist-test-slave-pww0 krb5kdc[4265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786334118, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.153.124@KRBTEST.COM for kudu/127.27.153.124@KRBTEST.COM
I20260810 03:55:18.242664 4526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 701751cd74534d82837beba49375b4c4: No bootstrap required, opened a new log
W20260810 03:55:18.244896 4695 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:55:18.244988 4695 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:55:18.246841 4540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.234865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48078 (local address 127.27.153.125:45685)
0810 03:55:18.234970 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.234973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.235155 (+ 182us) server_negotiation.cc:407] Connection header received
0810 03:55:18.235267 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.235270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.235312 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.235374 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.236178 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.236984 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.238152 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.238481 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.238550 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:18.242285 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:18.242289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:18.242292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:18.242319 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:18.244124 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.244660 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.244664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.244666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.244771 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.245093 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.245096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.245098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.245342 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:18.245534 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:18.246515 (+ 981us) server_negotiation.cc:299] Negotiation successful
0810 03:55:18.246660 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 03:55:18.247130 4526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 701751cd74534d82837beba49375b4c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42855 } } peers { permanent_uuid: "701751cd74534d82837beba49375b4c4" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } } peers { permanent_uuid: "be6e19a2ca0f49ef9face5a36c093816" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } }
I20260810 03:55:18.247377 4526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 701751cd74534d82837beba49375b4c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 701751cd74534d82837beba49375b4c4, State: Initialized, Role: FOLLOWER
I20260810 03:55:18.247750 4526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 701751cd74534d82837beba49375b4c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42855 } } peers { permanent_uuid: "701751cd74534d82837beba49375b4c4" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } } peers { permanent_uuid: "be6e19a2ca0f49ef9face5a36c093816" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } }
I20260810 03:55:18.248286 4700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 701751cd74534d82837beba49375b4c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42855 } } peers { permanent_uuid: "701751cd74534d82837beba49375b4c4" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } } peers { permanent_uuid: "be6e19a2ca0f49ef9face5a36c093816" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } } }
I20260810 03:55:18.248435 4700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 701751cd74534d82837beba49375b4c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:55:18.249960 4526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 701751cd74534d82837beba49375b4c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:55:18.251272 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.235221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.153.124:42387 (local address 127.0.0.1:60384)
0810 03:55:18.235613 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0810 03:55:18.235628 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 03:55:18.235742 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 03:55:18.236019 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 03:55:18.236022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 03:55:18.236183 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0810 03:55:18.236272 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:18.236279 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.237212 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.237215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:18.238213 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 03:55:18.238224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.239561 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.239564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 03:55:18.239654 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:18.240106 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 03:55:18.240122 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 03:55:18.242086 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 03:55:18.249221 (+ 7135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:18.249224 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:18.249227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:18.249480 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:18.249777 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 03:55:18.249780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 03:55:18.249782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 03:55:18.249867 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 03:55:18.250732 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 03:55:18.250736 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 03:55:18.250850 (+ 114us) client_negotiation.cc:769] Sending connection context
0810 03:55:18.250954 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 03:55:18.251114 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280}
I20260810 03:55:18.252511 28261 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 03:55:18.252712 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.235434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60384 (local address 127.27.153.124:42387)
0810 03:55:18.235575 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.235580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.235676 (+ 96us) server_negotiation.cc:407] Connection header received
0810 03:55:18.235831 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.235834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.235868 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.235909 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.236372 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.237085 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.238312 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.238616 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.238680 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:18.242544 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:18.242548 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:18.242550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:18.242575 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:18.249022 (+ 6447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.249585 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.249588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.249590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.249672 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.249951 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.249953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.249955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.250173 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:18.250264 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:18.252415 (+ 2151us) server_negotiation.cc:299] Negotiation successful
0810 03:55:18.252551 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 03:55:18.253845 4353 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38246
I20260810 03:55:18.254143 4353 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 03:55:18.254276 4353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:55:18.245784 4290 master_runner.cc:186] Error getting master registration for 127.27.153.125:45685: OK, instance_id { permanent_uuid: "701751cd74534d82837beba49375b4c4" instance_seqno: 1786334117963529 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 03:55:18.255757 4290 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 03:55:18.256047 4657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be6e19a2ca0f49ef9face5a36c093816: Bootstrap starting.
I20260810 03:55:18.256225 4353 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42855 } } peers { permanent_uuid: "701751cd74534d82837beba49375b4c4" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } } peers { permanent_uuid: "be6e19a2ca0f49ef9face5a36c093816" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } }
I20260810 03:55:18.257651 4353 leader_election.cc:288] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9 [CANDIDATE]: Term 1 election: Requested vote from peers 701751cd74534d82837beba49375b4c4 (127.27.153.125:45685), be6e19a2ca0f49ef9face5a36c093816 (127.27.153.124:42387)
I20260810 03:55:18.258327 4476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "701751cd74534d82837beba49375b4c4"
I20260810 03:55:18.258613 4476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 701751cd74534d82837beba49375b4c4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:55:18.259059 4657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be6e19a2ca0f49ef9face5a36c093816: Neither blocks nor log segments found. Creating new log.
I20260810 03:55:18.260128 4657 log.cc:824] T 00000000000000000000000000000000 P be6e19a2ca0f49ef9face5a36c093816: Log is configured to *not* fsync() on all Append() calls
I20260810 03:55:18.260290 4476 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 701751cd74534d82837beba49375b4c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4e6de7ee2f74ee39569b2e6039c00d9 in term 1.
I20260810 03:55:18.260828 4314 leader_election.cc:302] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 701751cd74534d82837beba49375b4c4, d4e6de7ee2f74ee39569b2e6039c00d9; no voters:
I20260810 03:55:18.261350 4673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 03:55:18.261587 4673 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9 [term 1 LEADER]: Becoming Leader. State: Replica: d4e6de7ee2f74ee39569b2e6039c00d9, State: Running, Role: LEADER
I20260810 03:55:18.261907 4673 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42855 } } peers { permanent_uuid: "701751cd74534d82837beba49375b4c4" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } } peers { permanent_uuid: "be6e19a2ca0f49ef9face5a36c093816" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } }
W20260810 03:55:18.262353 4720 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 701751cd74534d82837beba49375b4c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:55:18.262516 4720 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:55:18.262694 4657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be6e19a2ca0f49ef9face5a36c093816: No bootstrap required, opened a new log
I20260810 03:55:18.263692 4673 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4e6de7ee2f74ee39569b2e6039c00d9. Latest consensus state: current_term: 1 leader_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42855 } } peers { permanent_uuid: "701751cd74534d82837beba49375b4c4" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } } peers { permanent_uuid: "be6e19a2ca0f49ef9face5a36c093816" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } } }
I20260810 03:55:18.263857 4673 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9 [sys.catalog]: This master's current role is: LEADER
I20260810 03:55:18.266861 4657 raft_consensus.cc:348] T 00000000000000000000000000000000 P be6e19a2ca0f49ef9face5a36c093816 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42855 } } peers { permanent_uuid: "701751cd74534d82837beba49375b4c4" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } } peers { permanent_uuid: "be6e19a2ca0f49ef9face5a36c093816" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } }
I20260810 03:55:18.267118 4657 raft_consensus.cc:729] T 00000000000000000000000000000000 P be6e19a2ca0f49ef9face5a36c093816 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be6e19a2ca0f49ef9face5a36c093816, State: Initialized, Role: FOLLOWER
I20260810 03:55:18.267513 4657 consensus_queue.cc:260] T 00000000000000000000000000000000 P be6e19a2ca0f49ef9face5a36c093816 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42855 } } peers { permanent_uuid: "701751cd74534d82837beba49375b4c4" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } } peers { permanent_uuid: "be6e19a2ca0f49ef9face5a36c093816" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } }
I20260810 03:55:18.269043 4606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "be6e19a2ca0f49ef9face5a36c093816"
I20260810 03:55:18.269306 4606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be6e19a2ca0f49ef9face5a36c093816 [term 0 FOLLOWER]: Advancing to term 1
I20260810 03:55:18.271024 4606 raft_consensus.cc:2454] T 00000000000000000000000000000000 P be6e19a2ca0f49ef9face5a36c093816 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4e6de7ee2f74ee39569b2e6039c00d9 in term 1.
I20260810 03:55:18.269022 4657 sys_catalog.cc:564] T 00000000000000000000000000000000 P be6e19a2ca0f49ef9face5a36c093816 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 03:55:18.268119 4409 master_runner.cc:186] Error getting master registration for 127.27.153.124:42387: OK, instance_id { permanent_uuid: "be6e19a2ca0f49ef9face5a36c093816" instance_seqno: 1786334118192656 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 03:55:18.271804 4409 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 03:55:18.276728 4758 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be6e19a2ca0f49ef9face5a36c093816: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 03:55:18.276880 4758 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 03:55:18.279870 4736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 03:55:18.280956 4736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 03:55:18.282157 4540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.264137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48094 (local address 127.27.153.125:45685)
0810 03:55:18.264263 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.264267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.264282 (+ 15us) server_negotiation.cc:407] Connection header received
0810 03:55:18.264321 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.264325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.264374 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.264451 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.265488 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.266297 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.272640 (+ 6343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.272994 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.273122 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:18.276551 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:18.276555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:18.276558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:18.276590 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:18.278525 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.279078 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.279082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.279084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.279184 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.279518 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.279522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.279524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.279760 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:18.279927 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:18.281856 (+ 1929us) server_negotiation.cc:299] Negotiation successful
0810 03:55:18.281982 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 03:55:18.283121 4741 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6e19a2ca0f49ef9face5a36c093816 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42855 } } peers { permanent_uuid: "701751cd74534d82837beba49375b4c4" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } } peers { permanent_uuid: "be6e19a2ca0f49ef9face5a36c093816" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } } }
I20260810 03:55:18.283272 4741 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6e19a2ca0f49ef9face5a36c093816 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:55:18.287916 4606 raft_consensus.cc:1261] T 00000000000000000000000000000000 P be6e19a2ca0f49ef9face5a36c093816 [term 1 FOLLOWER]: Refusing update from remote peer d4e6de7ee2f74ee39569b2e6039c00d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:55:18.288561 4476 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 701751cd74534d82837beba49375b4c4 [term 1 FOLLOWER]: Refusing update from remote peer d4e6de7ee2f74ee39569b2e6039c00d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 03:55:18.288604 4673 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be6e19a2ca0f49ef9face5a36c093816" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:55:18.289148 4673 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "701751cd74534d82837beba49375b4c4" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 03:55:18.292779 4700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 701751cd74534d82837beba49375b4c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4e6de7ee2f74ee39569b2e6039c00d9. Latest consensus state: current_term: 1 leader_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42855 } } peers { permanent_uuid: "701751cd74534d82837beba49375b4c4" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } } peers { permanent_uuid: "be6e19a2ca0f49ef9face5a36c093816" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } } }
I20260810 03:55:18.292783 4741 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6e19a2ca0f49ef9face5a36c093816 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4e6de7ee2f74ee39569b2e6039c00d9. Latest consensus state: current_term: 1 leader_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42855 } } peers { permanent_uuid: "701751cd74534d82837beba49375b4c4" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } } peers { permanent_uuid: "be6e19a2ca0f49ef9face5a36c093816" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } } }
I20260810 03:55:18.292917 4700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 701751cd74534d82837beba49375b4c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:55:18.292918 4741 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6e19a2ca0f49ef9face5a36c093816 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:55:18.295197 4405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.263387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38256 (local address 127.27.153.126:42855)
0810 03:55:18.263520 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.263524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.280802 (+ 17278us) server_negotiation.cc:407] Connection header received
0810 03:55:18.280850 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.280853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.280903 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.280967 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.281802 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.282510 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.284070 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.284430 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.284520 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:18.286206 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:18.286210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:18.286213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:18.286244 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:18.287918 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.293473 (+ 5555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.293476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.293479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.293582 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.293896 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.293900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.293901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.294163 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:18.294277 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:18.294889 (+ 612us) server_negotiation.cc:299] Negotiation successful
0810 03:55:18.295020 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 03:55:18.297254 4663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:18.264468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60390 (local address 127.27.153.124:42387)
0810 03:55:18.264572 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.264576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.287738 (+ 23162us) server_negotiation.cc:407] Connection header received
0810 03:55:18.287996 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.287999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.288049 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.288098 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.289205 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.289912 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.291099 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:18.291417 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:18.291515 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:18.293397 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 03:55:18.293401 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 03:55:18.293404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 03:55:18.293432 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 03:55:18.295084 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.295844 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.295848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.295850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.295942 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 03:55:18.296254 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 03:55:18.296258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 03:55:18.296260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 03:55:18.296480 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 03:55:18.296584 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:18.296980 (+ 396us) server_negotiation.cc:299] Negotiation successful
0810 03:55:18.297099 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 03:55:18.302234 4673 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42855 } } peers { permanent_uuid: "701751cd74534d82837beba49375b4c4" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } } peers { permanent_uuid: "be6e19a2ca0f49ef9face5a36c093816" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } } }
I20260810 03:55:18.302385 4673 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9 [sys.catalog]: This master's current role is: LEADER
I20260810 03:55:18.302776 4762 mvcc.cc:205] Tried to move back new op lower bound from 7316824548497788928 to 7316824548403945472. Current Snapshot: MvccSnapshot[applied={T|T < 7316824548497788928}]
I20260810 03:55:18.302738 4741 sys_catalog.cc:455] T 00000000000000000000000000000000 P be6e19a2ca0f49ef9face5a36c093816 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42855 } } peers { permanent_uuid: "701751cd74534d82837beba49375b4c4" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } } peers { permanent_uuid: "be6e19a2ca0f49ef9face5a36c093816" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } } }
I20260810 03:55:18.302852 4741 sys_catalog.cc:458] T 00000000000000000000000000000000 P be6e19a2ca0f49ef9face5a36c093816 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:55:18.303383 4763 mvcc.cc:205] Tried to move back new op lower bound from 7316824548497788928 to 7316824548403945472. Current Snapshot: MvccSnapshot[applied={T|T < 7316824548497788928}]
I20260810 03:55:18.307364 4673 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42855 } } peers { permanent_uuid: "701751cd74534d82837beba49375b4c4" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } } peers { permanent_uuid: "be6e19a2ca0f49ef9face5a36c093816" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } } }
I20260810 03:55:18.307689 4673 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9 [sys.catalog]: This master's current role is: LEADER
I20260810 03:55:18.308583 4700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 701751cd74534d82837beba49375b4c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4e6de7ee2f74ee39569b2e6039c00d9" member_type: VOTER last_known_addr { host: "127.27.153.126" port: 42855 } } peers { permanent_uuid: "701751cd74534d82837beba49375b4c4" member_type: VOTER last_known_addr { host: "127.27.153.125" port: 45685 } } peers { permanent_uuid: "be6e19a2ca0f49ef9face5a36c093816" member_type: VOTER last_known_addr { host: "127.27.153.124" port: 42387 } } }
I20260810 03:55:18.308708 4700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 701751cd74534d82837beba49375b4c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 03:55:18.310580 4736 catalog_manager.cc:1347] Generated new cluster ID: 5f190a59c28a4475a5f19ac4d9a40bd6
I20260810 03:55:18.310642 4736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 03:55:18.323293 4736 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 03:55:18.324672 4736 catalog_manager.cc:1504] Loading token signing keys...
I20260810 03:55:18.338637 4736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4e6de7ee2f74ee39569b2e6039c00d9: Generated new TSK 0
I20260810 03:55:18.339753 4736 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 03:55:18.379619 4540 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:18.376024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48110 (local address 127.27.153.125:45685)
0810 03:55:18.376160 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.376165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.378385 (+ 2220us) server_negotiation.cc:407] Connection header received
0810 03:55:18.378595 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.378599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.378664 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.378717 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.379474 (+ 757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48110: BlockingRecv error: recv got EOF from 127.0.0.1:48110 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 03:55:18.381934 4663 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:18.377758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60400 (local address 127.27.153.124:42387)
0810 03:55:18.377869 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.377873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.380548 (+ 2675us) server_negotiation.cc:407] Connection header received
0810 03:55:18.380849 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.380853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.380907 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.380957 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.381796 (+ 839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60400: BlockingRecv error: recv got EOF from 127.0.0.1:60400 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 03:55:18.382917 4405 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:18.376905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38258 (local address 127.27.153.126:42855)
0810 03:55:18.377025 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.377029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.380211 (+ 3182us) server_negotiation.cc:407] Connection header received
0810 03:55:18.381098 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.381101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.381153 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.381205 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.382791 (+ 1586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38258: BlockingRecv error: recv got EOF from 127.0.0.1:38258 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 03:55:18.388151 4540 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:18.386187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48116 (local address 127.27.153.125:45685)
0810 03:55:18.386310 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.386315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.386908 (+ 593us) server_negotiation.cc:407] Connection header received
0810 03:55:18.387201 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.387204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.387296 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.387356 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.388030 (+ 674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48116: BlockingRecv error: recv got EOF from 127.0.0.1:48116 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260810 03:55:18.393482 4540 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:18.391862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48126 (local address 127.27.153.125:45685)
0810 03:55:18.391972 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.391976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.392522 (+ 546us) server_negotiation.cc:407] Connection header received
0810 03:55:18.392735 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.392739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.392792 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.392841 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.393362 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48126: BlockingRecv error: recv got EOF from 127.0.0.1:48126 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 03:55:18.399451 4540 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:18.397798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48138 (local address 127.27.153.125:45685)
0810 03:55:18.397908 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.397913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.398477 (+ 564us) server_negotiation.cc:407] Connection header received
0810 03:55:18.398687 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.398691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.398745 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.398795 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.399329 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48138: BlockingRecv error: recv got EOF from 127.0.0.1:48138 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 03:55:18.407500 4540 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:18.405778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48146 (local address 127.27.153.125:45685)
0810 03:55:18.405926 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.405930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.406482 (+ 552us) server_negotiation.cc:407] Connection header received
0810 03:55:18.406695 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.406699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.406754 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.406805 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.407344 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48146: BlockingRecv error: recv got EOF from 127.0.0.1:48146 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260810 03:55:18.420085 4540 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:18.418107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48160 (local address 127.27.153.125:45685)
0810 03:55:18.418238 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.418242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.418943 (+ 701us) server_negotiation.cc:407] Connection header received
0810 03:55:18.419174 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.419178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.419238 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.419293 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.419967 (+ 674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48160: BlockingRecv error: recv got EOF from 127.0.0.1:48160 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 03:55:18.441128 4540 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:18.439043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48176 (local address 127.27.153.125:45685)
0810 03:55:18.439196 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.439201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.439957 (+ 756us) server_negotiation.cc:407] Connection header received
0810 03:55:18.440206 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.440211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.440273 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.440333 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.441001 (+ 668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48176: BlockingRecv error: recv got EOF from 127.0.0.1:48176 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260810 03:55:18.477085 4540 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:18.475842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48190 (local address 127.27.153.125:45685)
0810 03:55:18.475971 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.475976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.476208 (+ 232us) server_negotiation.cc:407] Connection header received
0810 03:55:18.476343 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.476346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.476400 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.476470 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.476963 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48190: BlockingRecv error: recv got EOF from 127.0.0.1:48190 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 03:55:18.544571 4540 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:18.543307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48194 (local address 127.27.153.125:45685)
0810 03:55:18.543528 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.543534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.543728 (+ 194us) server_negotiation.cc:407] Connection header received
0810 03:55:18.543820 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.543822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.543874 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.543957 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.544446 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48194: BlockingRecv error: recv got EOF from 127.0.0.1:48194 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260810 03:55:18.675928 4540 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:18.674678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48210 (local address 127.27.153.125:45685)
0810 03:55:18.674844 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.674849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.674978 (+ 129us) server_negotiation.cc:407] Connection header received
0810 03:55:18.675139 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.675142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.675196 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.675271 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.675802 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48210: BlockingRecv error: recv got EOF from 127.0.0.1:48210 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260810 03:55:18.935135 4540 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 03:55:18.934123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48216 (local address 127.27.153.125:45685)
0810 03:55:18.934294 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:18.934298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:18.934332 (+ 34us) server_negotiation.cc:407] Connection header received
0810 03:55:18.934477 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:18.934480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:18.934531 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:18.934603 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 03:55:18.935009 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48216: BlockingRecv error: recv got EOF from 127.0.0.1:48216 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 03:55:19.264096 4720 catalog_manager.cc:1259] Loaded cluster ID: 5f190a59c28a4475a5f19ac4d9a40bd6
I20260810 03:55:19.264194 4720 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 701751cd74534d82837beba49375b4c4: loading cluster ID for follower catalog manager: success
I20260810 03:55:19.265944 4720 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 701751cd74534d82837beba49375b4c4: acquiring CA information for follower catalog manager: success
I20260810 03:55:19.267207 4720 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 701751cd74534d82837beba49375b4c4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:55:19.278286 4758 catalog_manager.cc:1259] Loaded cluster ID: 5f190a59c28a4475a5f19ac4d9a40bd6
I20260810 03:55:19.278378 4758 catalog_manager.cc:1552] T 00000000000000000000000000000000 P be6e19a2ca0f49ef9face5a36c093816: loading cluster ID for follower catalog manager: success
I20260810 03:55:19.280184 4758 catalog_manager.cc:1574] T 00000000000000000000000000000000 P be6e19a2ca0f49ef9face5a36c093816: acquiring CA information for follower catalog manager: success
I20260810 03:55:19.281399 4758 catalog_manager.cc:1602] T 00000000000000000000000000000000 P be6e19a2ca0f49ef9face5a36c093816: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 03:55:19.453440 4896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:19.449342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48226 (local address 127.27.153.125:45685)
0810 03:55:19.449608 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:19.449613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:19.449680 (+ 67us) server_negotiation.cc:407] Connection header received
0810 03:55:19.449862 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:19.449864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:19.449992 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:19.450085 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:55:19.450382 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:19.450946 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:19.451888 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:19.452131 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:19.452231 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:19.452569 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:55:19.453013 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:55:19.453133 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:19.453197 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 03:55:19.453269 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":70,"threads_started":1}
I20260810 03:55:19.460495 4906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:19.456479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60412 (local address 127.27.153.124:42387)
0810 03:55:19.456704 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:19.456709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:19.456726 (+ 17us) server_negotiation.cc:407] Connection header received
0810 03:55:19.456948 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:19.456951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:19.457076 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:19.457175 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:55:19.457406 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:19.457961 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:19.458811 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:19.459054 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:19.459134 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:19.459567 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:55:19.460075 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:55:19.460184 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:19.460272 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 03:55:19.460333 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":64,"threads_started":1}
I20260810 03:55:19.467558 4906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:19.464170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60420 (local address 127.27.153.124:42387)
0810 03:55:19.464277 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:19.464282 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:19.464497 (+ 215us) server_negotiation.cc:407] Connection header received
0810 03:55:19.464577 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:19.464580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:19.464632 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:19.464701 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:55:19.464930 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:19.465509 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:19.466527 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:19.466747 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:19.466839 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:19.467096 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:55:19.467180 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:55:19.467249 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:19.467350 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 03:55:19.467401 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 03:55:19.468801 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:19.463738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.27.153.126:42855)
0810 03:55:19.464031 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:19.464037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:19.464155 (+ 118us) server_negotiation.cc:407] Connection header received
0810 03:55:19.464410 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:19.464413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:19.464601 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:19.464719 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:55:19.465042 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:19.465890 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:19.466875 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:19.467169 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:19.467277 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:19.468113 (+ 836us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:55:19.468412 (+ 299us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:55:19.468556 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:19.468592 (+ 36us) server_negotiation.cc:299] Negotiation successful
0810 03:55:19.468658 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":101,"threads_started":1}
I20260810 03:55:19.469066 4896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 03:55:19.463745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48234 (local address 127.27.153.125:45685)
0810 03:55:19.464291 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0810 03:55:19.464296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 03:55:19.464460 (+ 164us) server_negotiation.cc:407] Connection header received
0810 03:55:19.464667 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 03:55:19.464671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 03:55:19.464729 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 03:55:19.464794 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 03:55:19.465116 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:19.465892 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:19.467834 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 03:55:19.468116 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 03:55:19.468213 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 03:55:19.468559 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 03:55:19.468638 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 03:55:19.468680 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 03:55:19.468887 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 03:55:19.468947 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442}
I20260810 03:55:19.473539 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 4290
I20260810 03:55:19.495569 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 4409
I20260810 03:55:19.508921 28261 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM5UkAJ/build/debug/bin/kudu with pid 4544
2026-08-10T03:55:19Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3293 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6700 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (304654 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST